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Abst  ract 

The  purpose  of  this  study  was  to  provide  the 
inexperienced  quality  assurance  specialist  ( QAS )  with  a 
guide  to  quality  assurance  indicators  for  use  when  working 
with  contractors  in  the  defense  electronics  industry.  The 
quality  indicators  used  by  two  defense  industry  contractors 
(ITT  and  Texas  Instruments)  with  excellent  quality  programs 
were  studied,  as  were  the  indicators  used  by  two  of  the  AFSC 
product  divisions  ( ESD  and  A S D )  that  interface  with  the 
defense  electronics  industry. 

This  research  focused  on  the  elements  of  a  MIL-Q-S858A 
quality  program  required  for  the  complex  systems  produced  by 
the  contractors  studied.  Quality  indicators  and  observations 
about  quality  programs  were  discussed,  presenting  the  uses, 
merits,  and  shortfalls  of  the  elements  a  QAS  might  find  in  an 
"ideal"  quality  program. 


A  GUIDE  TO  QUALITY  ASSURANCE  INDICATORS  FOR  THE  DEFENSE 

ELECTRONICS  INDUSTRY 


I.  Introduction 


General  Issue 

The  defense  electronics  industry  constantly  strives  to 
provide  the  latest  technology  to  the  government  in  response 
to  increasingly  complex  defense  requirements.  This  has 
resulted  in  a  dynamic  industry,  one  which  modifies  its 
methods  and  resources  as  needed,  based  on  ever-changing 
technology.  This  change  is  notable  when  comparing  the 
technology  required  for  the  B-2  with  the  technology  in  even 
the  most  recently  upgraded  B-52.  The  electronic  warfare 
threat  and  the  requirements  to  counter  that  threat  advance 
with  each  new  generation  of  technology. 

Many  of  the  specifications  and  standards  which 
govern  the  development  of  these  high-tech  systems,  however, 
have  not  kept  pace  with  the  changing  technology.  The  most 
recent  revision  of  several  specifications  reveals  that  they 
are  older  than  many  of  the  individuals  who  rely  on  them 
daily.  A  newly  assigned  officer,  who  has  witnessed  the 
Apollo  moon  landing  as  well  as  the  advent  of  the  space 
shuttle,  has  seen  dramatic  changes  in  technology.  In 
spite  of  this  dynamic  environment,  the  specifications  he 
relies  on  to  insure  the  design,  production,  and  quality  of 
the  United  States'  next  generation  fighter  remain  unchanged. 
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The  potential  danger  of  aging  publications  is 
compounded  by  the  Air  Force's  attempt  to  develop  well- 
rounded  officers.  The  goal  is  officers  who  are  able  to 
handle  the  varied  environments  of  potential  assignments. 

One  technique  used  to  broaden  skills  is  the  transfer  of 
officers  to  a  new  position  every  one  to  four  years. 

Although  the  long-term  result  of  broadly  knowledgeable 
officers  is  achieved,  a  short-term  result  is  also  realized: 
officers  who  are  continually  learning  new  systems,  officers 
trying  to  do  their  best  in  an  unfamiliar  position.  These 
officers  need  guidance  on  what  to  do  and  how  to  do  it. 
Guidance,  although  readily  available  in  the  many 
regulations,  standards,  specifications,  and  handbooks 
published  by  the  Air  Force,  is  often  obscured  by  the  sheer 
magnitude  of  information  available.  The  volume  of  the 
information  alone  often  stands  as  a  barrier  to  learning. 

As  noted  above,  many  of  these  publications  have  been 
used  as  guidance  for  years,  often  for  more  than  two  decades. 
The  primary  publication  guiding  Quality  Assurance 
Specialists  (QASs),  MIL-Q-9858A,  "Quality  Program 
Requirements,"  was  revised  most  recently  in  December  1963. 
MIL-Q-9858A  is  intended  for  use  in  contracts  involving 
complex  end-items  (such  as  missiles  and  aircraft)  and  their 
major  components  (including  electronic  systems  and 
navigation  systems).  MIL-Q-9858A  requires  .  .  the 

establishment  of  a  quality  program  by  the  contractor  to 
assure  compliance  with  the  requirements  of  the  contract," 
(4:3)  and  requires  the  contractor  to  use  the  program  he  has 


2 


developed.  The  quality  program  developed  not  only  must 
ensure  the  quality  of  the  end  product,  but  must  also 

.  .  assure  adequate  controls  throughout  all  areas  of 

contract  performance;  e.g.,  development,  manufacturing,  and 
shipping"  (4:3). 

The  associated  "interpreting"  handbook  used  in 
conjunction  with  MIL-Q-9858A  is  Handbook  H50,  "Evaluation  of 
a  Contractor's  Quality  Program,"  which  was  most  recently 
revised  in  April  1965.  Handbook  H50  is  intended  for  use  by 
both  the  government  and  industry  to  clarify  the  intent  and 
applicability  of  the  specification.  Each  subsection  of  the 
specification  is  quoted,  followed  by  a  Review  of  the 
Requirement  (general  discussion  on  the  intent  of  the 
subsection),  Applications  (examples  of  how  the  requirement 
has  been  applied  in  the  past),  and  Criteria  for  Evaluation 
(criteria  in  the  form  of  questions  which  should  be  asked  by 
the  reader  in  evaluating  a  section  of  the  program). 

The  information  contained  within  these  publications  is 
intended  as  general  guidance  only.  The  dates  of  last 
revision,  however,  suggest  that  the  latest  technologies  and 
quality  philosophies  may  not  be  represented. 

The  issue  of  outdated  guidance  for  Quality  Assurance 
specialists  considers  only  one  dimension;  another  facet  of 
the  problem  is  that  Air  Force  contractors  are  required  to 
comply  with  these  guides.  The  result  is  Air  Force  officers 
evaluating  state-of-the-art  defense  contractors,  both  of 
whom  are  guidea  by  material  most  recently  revised  over  25 
years  ago. 
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If  the  defense  industry  were  being  well-managed  under 

such  guidance,  no  further  discussion  of  this  situation  would 

be  warranted.  One  does  not  have  to  look  far,  however,  to 

note  this  is  not  the  case.  A  recent  article  in  the  Air 

Force  Times  notes  "...  about  92  percent  of  parts  sampled 

at  a  major  Air  Force  supply  center  are  of  lower  quality  than 

was  paid  for  .  .  ."  (9:35).  Further,  a  House  Republican 

Research  Committee  report  states  that 

.  .  .  quality  experts  estimate  that  the  total  cost  of 

poor  quality,  or  the  cost  of  not  doing  the  right  things 
right  the  first  time,  is  20%  of  gross  sales  for 
manufacturing  companies  ....  {16r2i 

A  third  example  of  poor  defense  industry  management  is  seen 

in  a  General  Accounting  Office  report  on  BIB  parts  problems. 

The  report  cites  low  reliability,  false  test  failures,  and 

delinquent  deliveries  or  missing  delivery  schedules  as 

impediments  to  operating  the  Bl-B  (26:4-5).  The  emphasis  of 

the  examples  is  clear:  either  the  Air  Force,  the  defense 

contractors,  or  both  are  being  inadequately  guided  in  their 

efforts  to  provide  quality  products  to  the  government. 

Specific  Problem 

Quality  Assurance  Specialists  (QASs)  are  currently 
being  guided  by  a  25-year-old  spec i f i ca t i on .  While 
experienced  QASs  have  perhaps  developed  a  personal  toolbox 
of  guides  or  indicators  that  alert  them  to  problems  with  a 
contractor's  quality  program,  QASs  newly  assigned  to  quality 
assurance  do  not  have  this  luxury.  They  must  consult  the 
available  publications  for  guidance.  As  these  publications 
are  by  design  very  general  in  nature,  little  current, 
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spc-jific  guidance  is  available  for  measuring  the  status  of  a 
contractor's  quality  program.  This  thesis  will  address  this 
problem  by  presenting  a  guide  of  specific  indicators  that 
can  be  used  in  evaluating  defense  electronics  industry 
contractors . 

Research  Objectives 

This  study  will  summarize  quality  assurance  indicators 
used  by  two  defense  electronics  contractors  (ITT  and  Texas 
Instruments)  and  two  Air  Force  Systems  Command  product 
divisions  (Aeronautical  Systems  Division  and  Electronics 
System  Division)  into  a  guide  of  quality  indicators  usable 
by  a  newly  assigned  QAS .  The  guide  will  be  a  presentation 
of  measures  available  for  evaluating  the  status  of  the 
contractor's  quality  program  in  the  defense  electronics 
industry.  To  accomplish  this,  the  following  research 
questions  will  be  studied: 

1.  How  is  quality  defined  in  the  industry?  The 
quality  philosophy  used  will  influence  the  types  of 
indicators  used. 

2.  Which  areas  of  the  contractor's  quality  program 
should  be  emphasized  in  determining  the  status  of  the 
quality  program?  A  brief  review  of  MIL-Q-9858A  reveals 
the  many  areas  that  contribute  to  a  Quality  Program. 
These  areas  are  covered  in  the  Common  Indicators 
section  in  the  literature  review.  This  objective  will 
identify  those  areas  that  will  provide  the  best  overall 
status  of  the  contractor's  quality  program. 
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3.  What  quality  indicators  are  currently  being  used  by 
both  the  industry  and  the  government?  This  information 
will  provide  the  baseline  indicators  to  be  included  in 
the  guide.  A  listing  and  discussion  of  currently  used 
indicators  will  be  valuable  to  the  QAS  when  choosing 
the  quality  indicators  that  are  appropriate  for  his 
program. 

4.  What  information  is  used  as  a  basis  for  each 
indicator?  Although  two  locations  may  use  the  same 
indicator  (e.g.,  assembly  line  defects),  the 
information  base  may  differ  (e.g.,  defects  per  man-hour 
versus  defects  per  unit). 

5.  Who  uses  each  indicator,  and  why?  Collecting 
information  for  an  indicator  is  not  enough;  the 
indicator  must  be  used  for  it  to  be  effective.  This 
objective  will  identify  the  use  of,  and  organizational 
level  of  personnel  using,  each  indicator. 

6.  Which  portion  of  the  quality  status  does  each 
indicator  assess,  and  how  comprehensive  is  the 
assessment?  In  recognition  of  the  fact  that  no  single 
indicator  can  assess  the  entire  quality  status,  the 
coverage  of  each  indicator  will  be  discussed. 

7.  What  indicators  are  not  currently  being  used  by  the 
Air  Force  that  may  provide  a  better  assessment  of  the 
quality  status  of  the  contractor?  This  is  perhaps  the 
most  important  objective,  with  the  intent  being  to 
identify  those  indicators  being  used  by  industry  but 
not  being  evaluated  by  the  Air  Force. 
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Scope 


The  objective  of  this  thesis  is  to  research  and 
document  the  key  quality  assurance  indicators  used  in  the 
defense  electronics  industry  This  documentation  can  be 
used  by  newly  assigned  officers  in  both  the  Aeronautical 
Systems  Division  (ASD)  and  the  Electronics  Systems  Division 
( ESD )  of  the  Air  Force  Systems  Command  (AFSC)  who  interface 
with  defense  electronics  industry  contractors  in  procuring 
products  for  the  Air  Force.  The  resulting  indicators  will 
aid  the  QAS  in  evaluating  the  quality  status  of  the 
contractor . 
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II.  Review  of  Literature 
Organization  of  Review 

This  review  begins  with  a  discussion  of  quality  as 
presented  by  experts  in  the  field.  Following  this  will  be  a 
short  discussion  of  the  types  of  quality  problems  and  a 
review  of  the  purpose  and  uses  of  quality  indicators.  These 
sections  underscore  the  importance  of  quality  indicators  and 
examine  the  difficulty  of  evaluating  quality.  Finally,  a 
review  of  the  commonly  used  indicators  will  be  presented, 
including  those  required  by  the  Air  Force. 

Definitions  of  Quality 

The  dictionary  definition  of  quality  is  given  as  a 
"peculiar  and  essential  character"  and  a  "degree  of 
excellence"  (27:944).  These  definitions  are  what  the  common 
person  might  perceive  quality  to  be:  the  peculiar 
characteristics  of  an  object  that  show  a  degree  of 
excellence.  Although  widely  accepted,  this  definition  has 
great  shortcomings  when  attempted  to  be  used  to  measure 
qual i ty . 

Garvin  has  thoroughly  researched  the  quality  experts  of 
the  past  several  decades  and  presents  their  definitions  of 
quality,  as  well  as  a  definition  he  has  developed.  Garvin 
has  found  that  Crosby  refines  the  common  definition 
suggested  by  Webster  slightly  by  considering  quality  to  be 
conformance  to  requirements  (11:40-41).  With  this 
definition,  any  product  that  is  built  to  conformance  is 
considered  to  be  a  quality  product;  a  Rolex  watch  built  to 
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specification  is  a  quality  watch,  as  is  a  Timex  watch. 

Juran  views  quality  in  a  different  light,  defining  quality 
as  fitness  for  use  (11:41).  Thus,  a  Timex  may  be  considered 
lower  quality  than  a  Rolex  if  its  accuracy  or  reliability  is 
not  acceptable.  Juran' s  definition  can  be  reduced  to 
"quality  lies  in  the  eyes  of  the  beholder"  (11:41).  Tuchman 
returns  somewhat  to  the  common  definition  when  she 
references  quality  as  being  innate  excellence  (11:41). 

A  person  with  this  view  might  comment,  "I  may  not  be  able  to 
define  quality,  but  I  know  quality  when  I  see  it."  Garvin 
remarks  that  "excellence,  according  to  this  view,  is  both 
absolute  and  universally  recognizable"  (11:41). 

Garvin  categorizes  Crosby's  view  (conformance  to 
requirements)  as  being  a  manufacturing-based  definition, 
Juran's  (fitness  for  use)  as  a  user-based  definition,  and 
Tuchman's  (innate  excellence)  as  a  transcendent  approach 
(13:25-28).  He  also  acknowledges  two  other  views  of  quality 
in  his  research.  The  product-based  definition  views  quality 
as  a  very  distinct  set  of  variables  or  characteristics,  and 
the  product  with  more  of  these  characteristics  is  of  higher 
quality  (13:25-26).  Finally,  he  defines  the  value-based 
view  as  providing  the  most  product  at  the  best  price 
(13:28).  An  individual  with  this  final  view  would  only 
consider  a  Rolex  to  be  of  higher  quality  if  it  kept  time 
substantially  more  accurately  than  the  Timex  and  if  the 
difference  in  performance  was  worth  the  difference  in  price. 

Not  being  satisfied  with  any  of  the  above  views  of 
quality,  Garvin  has  presented  his  own  (11:41-42).  His  list 
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of  eight  "dimensions  of  quality"  are:  performance, 
serviceability,  reliability,  conformance,  durability, 


features,  aesthetics,  and  perceived  quality.  "Each 

dimension  is  self-contained  and  distinct,  for  a  product  can 

be  high  on  one  dimension  while  being  low  on  another" 

(11:42).  In  support  of  his  definition,  Garvin  states: 

These  eight  dimensions  of  quality  incorporate  all  three 
of  the  traditional  definitions  [conformance  to 
requirements,  fitness  for  use,  innate  excellence],  as 
well  as  a  number  of  other  elements.  In  addition  to 
being  more  inclusive,  the  framework  suggests  an 
important  strategic  consideration  that  might  otherwise 
be  overlooked:  one  can  compete  on  quality  in  a  number 
of  different  ways.  (11:42) 

Takeuchi  and  Quelch  present  their  views  on  quality  by 
informing  the  xeader  that  "companies  must  be  sure  they  are 
offering  the  benefits  customers  seek.  Quality  should  be 
primarily  customer-driven,  not  technology-driven, 
production-driven,  or  competitor-driver."  (24:140). 

While  Garvin,  Juran,  Crosby,  Tuchman,  and  Takeuchi  each 
have  their  own  views  on  what  quality  is,  the  Air  Force  also 
has  a  definition:  the  composite  of  material  attributes, 
including  performance  features  and  characteristics,  of  a 
product  or  service  to  satisfy  a  given  need  (5:Encl  2).  This 
definition,  like  Garvin's,  incorporates  several  dimensions. 
The  user-based  (fitness  for  use)  approach  is  included,  as 
are  the  product-based  (distinct  set  of  characteristics)  and 
the  manufacturing-based  (conformance  to  requirements)  views. 
It  is  interesting  to  note  that,  in  spite  of  a  long-standing 
commercial  trend  to  the  contrary,  the  value-based  (most 
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product  at  the  best  price)  view  is  not  included  as  part  of 

the  official  Air  Force  definition. 

Peter  Angiola,  Staff  Assistant,  Defense  Industrial 

Productivity  and  Quality  Directorate,  expands  on  this 

official  Air  Force  definition: 

.  .  .  we  are  shifting  the  emphasis  on  making  a  product 

successful  as  viewed  from  conformance  to 
specifications,  to  conformance  to  correctly  defined 
specifications.  This  makes  the  contractors  responsible 
to  achieve  performance  objectives  rather  than  relying 
on  the  government  to  provide  detailed  specifications. 
This  will  hopefully  lead  to  systems  that  meet 
performance  objectives  and  mission  needs  rather  than 
simply  complying  to  specifications  for  legalistic 
reasons .  (16:9) 

In  the  terms  defined  by  Garvin,  this  definition  contains  the 
essence  of  Juran's  fitness  for  use  along  with  Crosby's 
conformance  to  requirements.  The  reader  should  note  the 
implication  of  the  phrase  "we  are  shifting"  as  used  above  is 
that  conformance  to  correctly  defined  specifications  is  not 
the  current  requirement,  but  a  goal.  This  is  not  so  much  a 
departure  from  the  Air  Force  requirement  that  the  product 
satisfy  the  specification  as  it  is  a  refinement  of  the 
requirement  to  put  the  user's  needs  first. 

MIL-Q-9858A  suggests  that  a  quality  product,  presumably 
as  defined  by  the  Air  Force  definition  given  above,  is  the 
end  product.  The  specification  does  not,  however,  limit  the 
contractor's  responsibilities  to  merely  providing  "quality" 
products  to  the  government.  A  system  of  acceptance  sampling 
is  closer  to  the  intent  of  MI L-I-4 5208A ,  Inspection  System 
Requirements,  which  is  applied  to  less  complex  items. 
MIL-I-45208A  relies  more  extensively  on  inspection  of  the 
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product  than  on  prevention  of  defects,  the  common  view  of 

quality  from  the  past  three  decades.  In  contrast,  the  more 

comprehensive  MIL-Q-9858A  requires  the  contractor  to  have 

.  .  .  an  effective  and  economical  quality  program, 

planned  and  developed  in  consonance  with  the 
contractor's  other  administrative  and  technical 
programs  .  .  .  [assuring]  .  .  .  adequate  quality 

throughout  all  areas  of  contract  performance.  .  .  . 

(6:1) 

The  requirement  outlined  above  provides  a  starting 
point  for  defining  what  the  Air  Force  expects  a  quality 
program  to  be.  First,  the  program  must  be  effective. 

Through  use  of  the  policies  and  practices  of  the  quality 
program,  the  contractor  should  be  able  to  ensure  every 
deliverable  meets  all  contract  requirements.  Second,  the 
program  should  be  cost  effective.  Much  has  been  written 
about  the  costs  of  quality,  and  Crosby  bases  his  quality 
philosophy  on  the  fact  that  quality  is,  in  fact,  free.  The 
government  neither  asks  nor  expects  the  contractor  to  spend 
a  great  deal  of  money  on  a  program  that  will  not  benefit  the 
contractor  accordingly.  Finally,  the  quality  program  should 
not  be  an  organizational  orphan.  The  responsibility  for 
quality  does  not  belong  solely  to  the  office  with  the  title 
"Quality  Assurance  Department."  To  have  a  truly  effective 
program,  quality  must  pervade  the  organizational  culture  of 
all  operations  and  functions  within  the  company. 

As  is  the  case  with  most  general  definitions,  the  Air 
Force  definition  specifies  only  the  general  characteristics 
a  quality  program  should  include.  No  attempt  has  been  made 
to  say  exactly  what  constitutes  a  quality  program.  No 
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listing  is  provided  which,  when  fulfilled  exactly,  would 
result  in  a  quality  program.  The  absence  of  a  formal 
definition  of  a  quality  program  challenges  both  the  QAS  and 
the  contractor  to  evaluate  the  status  of  the  quality 
program . 

General  Hansen,  Commander  of  Air  Force  Logistics 

Command,  has  provided  a  definition  of  quality  as  it  applies 

to  his  organization:  "We  must  have  the  right  goods  and 

services — spare  parts,  engineering,  and  maintenance--in  the 

right  places  at  the  right  times  .  .  (15:9).  Basically  a 

fitness  for  use  philosophym  this  AFLC  definition  of  quality 

partially  reflects  the  implied  Air  Force  definition  given 

above,  but  tailors  the  definition  to  fit  the  mission  of 

AFLC.  Application  of  any  quality  definition,  whether  that 

definition  was  provided  by  the  Air  Force  or  by  one  of  the 

quality  gurus,  may  be  unique  for  each  organization. 

Despite  the  absence  of  a  formal  definition  of  a  quality 

program,  measurement  of  that  program  is  essential.  As  Lord 

Kelvin  noted  over  a  hundred  years  ago: 

.  .  .  when  you  can  measure  what  you  are  speaking  about 

and  express  it  in  numbers,  you  know  something  about  it; 
but  when  you  cannot  express  it  in  numbers,  your 
knowledge  is  of  a  meagre  and  unsatisfactory  kind;  it 
may  be  the  beginning  of  knowledge,  but  you  have 
scarcely,  in  your  thoughts,  advanced  to  the  stage  of 
science,  whatever  the  matter  may  be.  (1:23) 

This  view,  hardly  a  recently  developed  philosophy,  justifies 

and  almost  requires  a  formal  definition  of  some  kind  be 

given  to  the  contractor's  quality  program  status.  If  the 

quality  program  status  cannot  be  measured,  knowledge  about 
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that  status  will  continue  to  be  "of  a  meagre  and 
unsatisfactory  kind." 


Quality  Problems 

In  a  survey  of  "many"  Fortune  500  companies,  Leonard 

and  Sasser  identified  the  following  perceived  sources  of 

quality  problems  (22:250): 

Workmanship/work  force 
Materials/purchase  of  parts 
Maintenance  of  process  equipment 
Design  of  process  equipment 
Product  design 
Control  systems 
Management 

Garvin  came  up  with  a  similar  list  in  his  research 
(12:660).  Each  of  these  areas  are  discussed  below  to 
familiarize  the  reader  with  the  types  and  diversity  of 
potential  quality  problems. 

The  quality  of  workmanship  suffers  when  the  individual 
"on  the  floor,"  the  one  actually  assembling  the  hardware, 
fails  to  follow  work  instructions  or  does  a  poor  job  in  a 
manual  process.  Such  quality  problems  can  range  from 
forgetting  to  install  a  washer  to  shorting  a  circuit  with  a 
poor  solder  joint. 

Materials  or  purchased  parts  that  don't  meet  standards 
ultimately  affect  the  quality  of  the  end  product.  A  worker 
installing  resistors  on  a  circuit  board  must  assume  the 
parts  supplied  to  him  are  suitable  for  the  job.  Regardless 
of  the  precision  with  which  the  resistor  is  mounted  to  the 
board,  if  the  resistor  doesn't  perform  properly,  the  quality 
of  the  end  product  will  suffer. 


Process  equipment  that  has  been  pressed  into  extra 
service  at  the  expense  of  routine  maintenance  is  not  likely 
to  control  the  process  at  the  required  leve^.  If  the 
accuracy  of  the  process  is  to  be  the  point  of  evaluation 
instead  of  final  product  inspection,  the  result  can  be  a 
high  proportion  of  nonconforming  parts. 

The  design  of  the  process  equipment  itself  can 
contribute  to  the  quality  of  the  final  product,  or  the  lack 
thereof.  Expecting  a  process  to  maintain  tolerances  beyond 
its  capability  is  an  invitation  to  quality  problems;  100% 
inspection  becomes  the  only  recourse,  a  poor  method  of 
attempting  to  guarantee  quality. 

The  design  of  the  product  must  allow  for  the 
manufacturing  and  quality  techniques  used  on  the  shop  floor 
rather  than  those  used  in  an  engineering  lab.  Differing 
skill  levels  as  well  as  differing  conditions  (time, 
lighting,  humidity,  temperature,  available  tools,  etc.)  must 
be  considered. 

Control  systems  are  designed  to  insure  the  final 
product  meets  requirements.  A  control,  such  as  the  drawing 
release  system  used  to  release  current,  approved 
manufacturing  and  assembly  drawings  to  the  floor,  can 
prevent  the  end  item  from  meeting  specification  if  the 
system  is  prone  to  error  or  slower  than  the  process. 

Finally,  management  can  be  the  direct  cause  of  poor 
product  quality;  studies  have  shown  that  management  can 
control  about  80%  of  defects  (20:315).  Being  ultimately 
responsible  for  the  decisions  leading  to  the  production  of 
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end  items,  they  are  also  responsible  if  the  wrong  items  are 
produced.  An  argument  can  be  made  that  management  is 
responsible  for  each  of  the  areas  listed  above,  but  such  a 
categor ization  may  amount  to  little  more  than  finger 
pointing  and  can  hinder  the  resolution  of  problems. 

With  such  a  diversified  list  of  perceived  quality 
problems,  from  design  to  process  to  management,  it  is 
evident  that  no  single  indicator  can  adequately  assess  the 
status  of  a  contractor's  quality  program. 

Common  Indicators 

The  Air  Force  requires  several  types  of  indicators  in 

MIL-Q-9858A,  Quality  Program  Requirements.  Among  these  are 

measures  of  the  quality  and  quality  costs  of  purchased 

material,  subcontracted  work,  drawings,  engineering  changes, 

measuring  equipment,  manufacturing  work,  fabrication  work, 

and  assembly  work  (6:1).  Inspection  and  testing  records 

must  be  kept,  indicating  types  and  numbers  of  deficiencies, 

and  the  action  taken  in  connection  with  those  deficiencies 

(6:3).  Handbook  H50,  the  guide  to  be  used  in  conjunction 

with  MIL-Q-9858A,  also  suggests  that  the  records  "indicate 

the  percentage  of  items  passing  inspection  or  test  and  the 

quantities  of  acceptable  and  rejected  items,"  and  requires 

records  be  kept  of  "compliance  or  noncompliance  with  work 

instructions"  (4:10).  Handbook  H50  further  recommends  that: 

.  .  .  records  of  subcontractor  quality  assurance 

programs,  of  engineering  approvals,  of  customer  returns 
and  cost  records  pertinent  to  acceptance  of  non- 
conforming  materials  [as]  examples  of  the  records 
required  for  an  effective  quality  program.  (4:10) 
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Finally,  MIL-Q-9858A  requires  the  contractor  to  have 
available  "data  associated  with  the  costs  and  losses  in 
connection  with  scrap  and  with  rework  necessary  to  reprocess 
nonconforming  material  to  make  it  conform  completely"  (6:7). 

The  indicators  identified  above  are  from  MIL-Q-9858A,  a 
document  intended  for  use  as  a  guide  to  both  the  contractor 
and  the  QAS .  The  diversity  and  volume  of  indicators 
suggested  or  required  by  the  government  can  be  intimidating 
to  the  new  QAS  trying  to  determine  the  best  way  to  evaluate 
a  contractor. 

Purpose  and  Use  of  Indicators 

MI L-Q-9 8 58A  clearly  states  one  of  the  purposes  of 
maintaining  quality  assurance  indicators:  "Management 
regularly  shall  review  the  status  and  adequacy  of  the 
quality  program"  (6:2).  This  direction  emphasizes  that  the 
existence  of  a  quality  program  is  not  sufficient;  the 
quality  program  must  be  a  dynamic  process  subject  to  change 
as  necessary. 

Thinking  of  quality  as  a  process  ties  strongly  to  one 
of  the  current  philosophies  of  quality:  design  and 
manufacture  quality  into  the  product  instead  of  trying  to 
inspect  quality  into  the  product.  This  philosophy  suggests 
the  idea  of  controlling  the  process  by  which  products  are 
manufactured  or  developed.  Exemplified  by  statistical 
process  control,  controlling  the  process  within  acceptable 
limits  will  help  insure  consistent  product  quality. 

Likewise,  controlling  the  quality  program  within  acceptable 
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limits  (its  status  and  adequacy)  will  also  help  insure 

consistent  product  quality.  The  means  to  assuring  the 

adequacy  of  the  contractor's  quality  program,  according  to 

MIL-Q-9858A,  is  the  "use  (of]  any  records  or  data  essential 

to  the  economical  and  effective  operation  of  his  quality 

program"  (6:3).  The  handbook  used  in  conjunction  with 

MI L-Q-98 58A ,  Handbook  H50,  places  the  responsibility  for  an 

adequate  quality  program  squarely  on  the  contractor: 

The  contractor  is  solely  responsible  for  the  control  of 
product  quality  and  for  offering  to  the  government  for 
acceptance  only  products  determined  by  him  to  conform 
to  contractual  requirements.  (4:1) 

Handbook  H50  continues,  however,  by  stating  that  "the 
government's  evaluation  plan  should  apply  to  all  aspects  of 
a  contractor's  program"  (4:1).  Clearly  the  government  is 
responsible  for  evaluating  the  contractor's  quality  status; 
such  an  evaluation  is  best  performed  using  the  tools 
developed  by  those  most  knowledgeable  about  contractor 
quality  programs:  individuals  in  the  contractor's  quality 
program  office.  While  using  the  contractor's  tools  does  not 
relieve  the  QAS ' s  responsibility  of  verifying  their 
appropriateness  as  quality  status  measures,  it  does  prevent 
the  QAS  from  reinventing  the  wheel. 

The  uses  of  indicators  are  as  diverse  as  the  types  of 
indicators.  MIL-Q-9858A  requires  "analysis  of  trends  in 
processes  or  performance  of  work  to  prevent  nonconforming 
product"  (6:3).  Quality  cost  data  "shall  serve  the  purpose 
of  identifying  the  cost  of  both  the  prevention  and 
correction  of  nonconforming  supplies"  (6:3)  and  to 
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"facilitate  sound  decision  making  by  contractors  regarding 
their  quality  program"  (4:12).  "Adequate  and  economical 
control  of  [purchased]  material"  (6:4-5)  is  required,  as  is 
"final  inspection  and  test  of  completed  products  [to] 
provide  a  measure  of  the  overall  quality  of  the  completed 
product"  (6:6).  A  Department  of  Defense  Directive  has  a 
requirement  that 

DoD  components  shall  develop  and  manage  a  quality 
program  to  achieve  the  following  objectives:  assure 
mission  and  operational  effectiveness  and  user 
satisfaction  with  DoD  products;  [and]  assure  that  all 
services  provided  and  products  designed,  developed, 
purchased,  [or]  produced  .  .  .  conform  to  specified 

requirements.  (5:1) 

This  requirement  expresses  the  use  of  a  quality  program 
"in-house"  as  well  as  at  the  contractor. 

Garvin  suggests  that  appropriate  indicators  might 
"capture  the  impact  of  incoming  materials  on  processing 
times  and  test  procedures"  (10:6).  This  concern  focuses  on 
the  fact  that  although  all  quality  problems  do  not  originate 
from  within  the  contractor's  plant,  the  contractor  is  not 
absolved  from  detecting  and  preventing  these  problems  as 
well. 


Review  Summary 

This  chapter  has  examined  several  aspects  of  quality 
assurance  indicators,  beginning  with  the  various  definitions 
of  quality.  Experts  have  defined  quality  in  many  ways, 
viewing  it  as  fitness  for  use,  conformance  to  requirements, 
value  for  the  money,  innate  excellence,  or  number  of 
attributes  present.  The  Air  Force  definition  of  quality 


includes  several  of  these  views.  The  contractors' 
definitions  and  measurements  of  quality  will  be  examined  in 
Chapter  IV. 

The  variety  of  quality  problems  facing  the  industry 
were  reviewed,  and  responsibility  for  defects  was  seen  to 
lie  not  solely  with  the  quality  department,  but 
primarily  with  management.  There  are  many  participants 
in  maintaining  top  quality,  including  workers,  the 
subcontracts  department,  design  engineers,  and  manufacturing 
engineers.  The  range  of  defects  provides  a  corresponding 
range  of  potential  indicators,  as  shown  in  the  discussion 
of  common  indicators.  The  areas  that  are  actually  monitored 
by  the  industry  and  by  the  government  will  be  reviewed  in 
the  next  chapters. 

The  purpose  of  quality  indicators  was  identified  as  a 
means  for  the  contractor  to  monitor  the  status  of  his 
quality  program;  the  uses  of  indicators  were  reviewed, 
concentrating  in  such  areas  as  trend  monitoring, 
quality-cost  evaluation,  and  purchased  material  appraisal. 

The  reader  must  keep  in  mind  that  the  defense 
electronics  industry  revolves  around  the  production  of 
complex  items,  not  capacitors  and  resistors.  In  many  cases, 
the  product  can  not  be  fully  tested  upon  completion;  only 
key  dimensions  of  the  product's  operability  can  be  evaluated 
at  final  test.  This  focus  requires  a  process  that  will 
insure  the  quality  of  the  end  product  and  not  one  that  will 
merely  attempt  to  determine  the  final  acceptability  of  that 
product . 
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The  following  chapters  will  discuss  how  two  contractors 
in  the  defense  electonics  industry  view  quality  and 
examine  the  emphasis  placed  on  the  various  aspects  of  their 
quality  programs.  Examples  of  the  indicators  used  will  be 
presented,  along  with  the  reasoning  for  those  indicators. 

The  government  view  of  quality  will  also  be  reviewed,  with 
examples  of  the  indicators  used  by  QASs.  Differing  degrees 
of  emphasis  on  the  importance  of  areas  of  the  quality 
program  is  expected  due  to  the  proximity  of  the  evaluators 
to  the  production  effort;  the  contractor  is  managing  the 
program  on  a  daily  basis,  while  the  QAS  normally  monitors 
the  program  on  a  weekly  or  monthly  basis. 
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III.  Methodolo 


This  chapter  describes  in  detail  the  methodology  used 
for  this  research.  First,  the  approach  used  will  be 
broadly  described,  followed  by  a  justification  for  that 
approach.  The  questions  used  to  guide  the  discussions  at 
each  location  will  be  covered  in  the  interview  guide. 
Finally,  the  population  selected,  the  information  collection 
plan,  and  the  method  of  information  analysis  will  be 
desc  r i bed . 

General  Approach 

The  following  broad  steps  were  used  to  reach  the 
objective  of  providing  the  new  QAS  with  a  guide  of  possible 
quality  assurance  indicators: 

1.  Obtain  quality  indicators,  by  interview,  from  both 
contractor  and  Air  Force  personnel. 

2.  Discuss  the  information  base  and  use  of  each 
indicator . 

3.  Propose  a  list  of  quality  indicators  based  on  what 
an  ideal  quality  program  might  contain. 

Justification  of  Approach 

Interviews  were  chosen  as  the  method  of  information 
gathering  for  several  reasons.  First,  interviews  provided 
the  best  opportunity  to  develop  a  guided,  but  unstructured, 
information  base.  The  guided  discussion  allowed  the 
inclusion  of  ideas  and  observations  not  originally 
considered  during  the  planning  of  the  interviews.  Second, 
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surveys  were  too  limited  in  scope  for  this  project,  and 
receiving  the  information  by  mail  presented  the  very  real 
probability  of  incomplete  information  being  submitted. 
Finally,  telephone  interviews  would  have  involved  an 
excessive  amount  of  time  on  the  part  of  the  interviewee, 
again  raising  the  probability  of  incomplete  information. 

Interview  Guide 

An  interview  guide  was  used  to  conduct  the  interviews 
due  to  the  open-ended  nature  of  the  information  desired. 

The  guide  included  the  questions  posed  in  the  research 
objectives,  stated  here  in  the  format  used  at  the 
contractor's  facility.  Several  of  the  questions  were  of 
general  use  in  obtaining  background  information  about  the 
contractor's  facilities  and  quality  program,  and  therefore 
do  not  address  the  issue  of  indicators  directly. 

1.  How  does  your  company  define  quality? 

2.  What  are  the  components  of  your  quality  program? 

3.  What  quality  indicators  are  used  by  your  company 

in  assessing  the  quality  status  of  a  program? 

a.  What  information  is  used  as  the  basis  for  each 
indicator? 

b.  At  what  organizational  level  is  each  indicator 
used? 

c.  Why/How  is  each  indicator  used? 

d.  How  adequate  is  the  assessment? 
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4.  What  are  the  goals  of  your  quality  program? 

a.  How  do  you  know/measure  when  these  goals  have 
been  achieved? 

b.  Is  continuous  quality  improvement  itself  used 
as  a  goal? 

5.  What  role  does  quality  play  in  the  design  process? 

a.  Design  for  quality  (design  of  experiments, 
robustness ) ? 

b.  Design  for  producibi 1 i ty? 

c.  Simplicity  of  inspection? 

6.  What  quality  tools  are  used  (Taguchi's  design  of 
experiments,  SPC,  fixture  control,  automated 
processes,  automated  inspection,  acceptance  sampling)? 

7.  How  is  customer  satisfaction  measured? 

8.  What  indicators  are  not  currently  required  by  the 
Air  Force  that  would  provide  a  better  assessment  of 
the  quality  status  of  the  contractor? 

Population 

The  intent  of  this  research  is  to  provide  an  index  of 
quality  indicators  as  currently  used  by  the  Air  Force  and 
selected  contractors,  not  to  provide  a  statistical  list  of 
indicators  most  commonly  used.  With  this  in  mind,  no 
attempt  was  made  to  gather  a  representative  sampling  of 
contractors  or  Air  Force  personnel. 

From  the  government  side,  AFSC/ASD  and  AFSC/ESD  were 
included  due  to  their  significant  business  with  the  defense 
electronics  industry.  The  2-letter  offices  responsible  for 
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quality  assurance  at  each  Division  (PM  at  ASD,  PL  at  ESD ) 
were  contacted  and  asked  to  prepare  a  presentation  of  the 
quality  indicators  and  quality  philosophies  used  at  their 
locations . 

From  the  industry  side,  only  two  contractors  (ITT 
and  Texas  Instruments)  were  chosen  due  to  time  and  fiscal 
constraints.  They  were  selected  based  on  personal 
experience  and  the  experience  of  individuals  in  ASD's 
Directorate  of  Manufacturing  and  Quality  Assurance  as 
companies  with  excellent  quality  programs.  The 
Vice-President  for  Quality  at  each  of  the  selected  companies 
was  asked,  first,  for  cooperation  and,  second,  to  prepare  a 
presentation  of  the  quality  indicators  and  quality 
philosophies  used  in  their  companies. 

As  expected,  industry  was  very  eager  to  participate  in 
this  study.  The  information  obtained  was  not  used  to  assess 
the  status  of  each  company's  quality  program,  but  to  provide 
a  guide  to  quality  indicators  for  the  new  QAS .  The  selected 
companies  were  proud  of  thei r  quality  programs  and  viewed 
this  study  as  an  opportunity  to  aid  the  Air  Force  in  its 
evaluation  efforts,  a  benefit  to  the  industry  when  newly 
assigned  QASs  are  involved. 

Data  Collection  Plan 

The  type  of  information  desired  involved  a  full  day  at 
each  location.  Use  of  a  tape  recorder  was  attempted, 
but  found  to  be  either  ineffective  (participants  were 
reluctant  to  speak  openly  while  the  recorder  was  running)  or 
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prohibitted  by  the  security  at  the  location.  Consequently, 
hand-written  notes  were  taken  during  the  interviews.  This 
method  of  collecting  information  was  certainly  not  the  most 
efficient,  but  was  necessitated  by  the  situation.  Each 
location  did  supplement  the  interview,  however,  by  supplying 
hardcopies  of  the  indicators  of  interest  (presented  in 
Chapter  IV ) . 

Analysi s 

As  noted  in  Chapter  I,  the  intent  of  this  thesis  is  to 
provide  a  guide  to  quality  indicators  to  be  used  by  officers 
newly  assigned  as  Quality  Assurance  Specialists  (QASs).  The 
nature  of  the  population  selected  and  the  intent  of  the 
study  dictated  that  a  statistical  evaluation  would  not  be 
beneficial.  Instead,  the  merits  and  shortfalls  of  the 
indicators  contributed  by  each  location  were  presented  and 
discussed.  Based  on  a  review  of  the  indicators'  merits  and 
shortfalls,  and  on  the  observations  offered  by  each 
location,  the  elements  of  an  ideal  quality  program  were 
presented.  The  elements  of  the  "ideal"  quality  program  are 
intended  to  serve  as  the  guide  to  quality  indicators. 

This  list  of  indicators  is  only  intended  to  serve  as  a 
starting  point  for  the  QAS .  The  reader  is  encouraged  to 
evaluate  the  indicators  and  discussions  presented  in  light 
of  his  or  her  current  program  requirements.  The  analysis 
includes  elements  an  ideal  quality  program  might  contain, 
based  on  the  quality  indicators  and  observations  presented 
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by  the  four  locations,  but  in  no  way  is  the  analysis 
intended  to  limit  the  QAS  to  the  indicators  presented. 
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IV.  Findings 


Organization  of  Findings 

This  chapter  presents  the  results  of  the  interviews 
with  quality  managers  from  ITT,  TI ,  ESD ,  and  ASD.  Since  the 
intent  of  this  thesis  is  to  provide  a  guide  to  quality 
assurance  indicators  and  not  to  evaluate  the  quality 
programs  of  the  contractors  or  product  divisions  studied, 
not  all  the  indicators  or  programs  used  at  a  location  will 
be  presented.  Identical  indicators  used  at  more  than  one 
location  will  only  be  discussed  once,  with  minor  variations 
to  be  discussed  in  Chapter  V.  This  is  not  intended  to 
diminish  the  importance  of  the  indicators  at  the  other 
locations,  but  rather  to  provide  a  concise  presentation  of 
the  wide  variety  of  indicators  used. 

Numerical  values  and  program  identifiers  have  been 
removed  from  the  tables  and  figures  contributed  by 
the  contractors.  Actual  values  are  proprietary  and  would  be 
of  no  use  to  the  QAS  deciding  on  which  indicators  to  use  for 
his  or  her  program.  The  indicators  presented  by  ESD  are 
based  on  fictitious  data  and  do  not  represent  the  status  of 
either  ITT  or  Texas  Instruments. 

Each  interview  presentation  will  begin  with  background 
information,  providing  not  only  some  information  about  the 
products  managed  at  the  location,  but  also  an  insight  into 
the  operating  philosophy  at  the  location.  The  next, 
discussion  focuses  on  the  quality  indicators  used  by  that 
location,  providing  the  reader  with  examples  of  quality 


assurance  indicators.  Finally,  observations  made  by  the 
respondents  in  the  area  of  quality  or  quality  management  are 
presented.  These  observations  should  help  the  QAS  in 
evaluating  the  indicators  presented. 

Detailed  discussion  and  analysis  of  the  information 
will  be  presented  in  Chapter  V. 

ITT  Avionics  Division  (ITTAV)  (23) 

Background .  ITTAV  is  a  division  of  ITT  Defense 
Corporation,  the  parent  organization  of  all  ITT  divisions 
doing  business  with  DoD.  Located  in  Nutlev,  New  Jersey 
three  programs  comprise  the  major  focus  at  ITTAV.  The 
largest  of  these  programs  is  the  AN/ALQ-165  Airborne 
Se 1 f -P r o tec t i on  Jammer  (ASPJ)  which  is  used  on  many 
airframes,  including  the  Air  Force  F-16.  The  second  major 
ITTAV  program  is  the  upgrade  and  production  cf  ALQ-136 
defensive  electronic  countermeasures  systems  for  use  on  the 
Army's  AH-64  Apache  helicopters,  the  OV-1,  and  potentially 
on  the  V-22  Osprey.  The  ALQ-172  jammer,  being  used  on  Air 
Force  B-52s  and  certain  C-130s,  is  the  final  major  program, 
and  is  nearing  the  last  years  of  its  production  buy.  The 
users  of  these  systems  have  expressed  their  satisfaction 
with  ITTAV's  products.  (14:23) 

ITTAV  defines  quality  as  conformance  to  specification, 
where  the  specifications  are  dictated  by  the  contractual 
relationship  between  the  government  and  ITTAV.  This 
definition  would  be  expected  since  conformance  to 
specification  is  what  they  are  paid  to  do.  Despite  this 
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required  definition,  ITTAV  attempts  to  identify  contractual 
inconsistencies,  deficiencies,  and  unreasonable 
requirements,  especially  at  the  system  level.  Should 
parameters  be  omitted  by  the  government,  or  be  at  a  level 
considered  too  lax  by  ITTAV,  the  company's  minimum  standards 
are  used  as  appropriate.  In  addition,  ITTAV  management 
advocates  the  "next  process  as  customer"  philosophy  for 
internal  operations. 

Quality  Indicators . 

Statistical  Process  Control.  ITTAV  employs 
several  methods  of  maintaining  quality  in  the  production 
environment,  some  of  which  can  also  be  defined  as  quality 
indicators.  One  of  these  methods  is  Statistical  Process 
Control  (SPC).  This  system  for  monitoring  the  status  of  a 
process  can  reveal  any  tendency  for  the  process  to  leave  the 
prescribed  norms,  ideally  allowing  corrective  action  to  be 
taken  before  the  product  is  affected. 

ITTAV  currently  has  twenty  applications  of  SPC, 
including  soldering  (wave,  hand,  bar,  and  vapor  phase), 
bench  assembly,  conformal  coating,  and  wirebonders.  The  SPC 
data  are  plotted  real-time  by  the  process  operators,  while 
analysis  of  the  SPC  charts  is  done  weekly  by  supervisors  and 
process  engineers  (the  operators  flag  undesirable  trends  if 
they  are  noted  during  the  week).  The  goals  in  this  area  are 
defined  by  the  upper  and  lower  control  limits  calculated  for 
the  SPC  chart.  Although  contractual  requirements  require 
100%  inspection  in  some  cases,  SPC  is  still  being  performed 
to  observe  trends  and  prevent  problems. 
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Design .  In  the  area  of  design,  ITTAV  has  a 
reliability,  maintainability,  safety  (RMS)  group  using  a 
Reliability,  Availability,  Maintainability  -  Computer  Aided 
Design  (RAM-CAD)  system  to  assist  in  the  design  of  circuits. 
Building  blocks  (circuits  common  to  particular  applications) 
are  designed  and  evaluated  for  RAM  values.  These  building 
blocks  are  then  evaluated  for  trends  in  RAM  values,  and 
field  data  are  correlated  with  lab  data  to  verify  the 
values.  No  specific  numeric  goals  are  set  in  this  area, 
rather  a  continuous  improvement  goal  is  implied. 

Vendor  Rating.  A  vendor  rating  system  reports 
monthly  on  the  quality  of  vendors  supplying  ITTAV.  The 
system  is  based  on  data  from  incoming  inspection,  source 
inspection,  survey  results  performed  by  the  Vendor  Quality 
group,  and  line  rejects  discovered  during  assembly  and  test. 
A  complexity  factor  is  assigned  based  on  the  unit  price  of 
the  commodity:  a  low  unit  price  results  in  the  highest 
complexity  factor  (1.0),  while  a  high  unit  price  results  in 
the  lowest  complexity  factor  (0.5).  The  actual  rating  for 
vendors  with  multiple  products  is  done  according  to  the 
following  formula 

LR ( A )  *  CF ( A )  +  LR ( B )  *  CF(B)  +  ...  +  LR ( n  )  *  CF(n) 

LI 

*  100  «  Rating  ( % )  ( 1 ) 

whe  re 

LR ( i )  *  lots  of  product  i  rejected 

CF(i)  «  complexity  factor  of  product  i 
LI  =  total  lots  inspected 
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Alphabetic  ratings  are  applied  to  vendors  based  on  the 

rating  percentages  as  determined  by  Eq  (1): 

Rating  A  (recommended)  -  5%  or  less 

Rating  B  (acceptable)  -  >  5%  but  <  15% 

Rating  C  (unsatisfactory)  -  15%  or  greater 

A  vendor  may  also  receive  an  automatic  C  rating  as  a  result 

of  unsatisfactory  survey  results,  failure  to  reply  to  a 

corrective  action  request  (CAR),  or  unsatisfactory  reply  to 

a  CAR . 

Summary  data  are  presented  in  two  formats.  Table  1 

Table  1.  Vendor  Ratings  by  Commodity  Category 

TC’A_  VENDORS  Fffi^ED 
MARCH,  1989 

A  V.  B  V.  C  V.  TQ  T  Ai_ 

mechanical  ASSE'-B-' 

ElEC’RICal  A3EE"B_v 
MECHANICAL  COMPONENTS 

Electrical  components 
raw  material 
MISCELLANEOUS 

total 


presents  the  data  by  commodity  category,  giving  the  number 
of  vendors  with  a  given  rating  as  well  as  the  percentage  of 
vendors  with  that  rating.  The  commodity  categories  were 
chosen  by  ITTAV  as  a  means  of  grouping  the  vendors  by  the 
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same  or  similar  types  of  items.  Figure  1  presents  the  data 
without  regard  to  commodity  category,  showing  only  the 
percentage  of  vendors  with  a  given  rating. 


Figure  1.  Monthly  Performance  of  Active  Vendors 

Qual i ty  Costs  .  An  ITTAV  report  details 

.  .  .  the  basic  concept  of  quality  costs  [as] 
recognition  and  organization  of  certain  costs  related 
to  quality  to  gain  knowledge  of  their  major 
contributing  segments  and  the  direction  of  their 
trends .  (18:1) 

To  this  end,  they  break  quality  costs  into  the  traditional 
categories  of  prevention,  appraisal,  and  failure. 

ITTAV-wide  total  quality  cost  data  is  presented  in  a 
monthly  report  as  a  cumulative  percentage  of  sales,  i.e., 
quality  costs  year-to-date  vs  sales  year-to-date,  with  a 
goal  set  by  management  based  on  past  performance  (Figure  2). 
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Figure  2. 


Quality  Costs  vs.  Sales,  Year-to-date 


This  data  is  also  tabulated  by  presenting  the  costs  in  the 
traditional  categories,  as  well  as  subcategories,  and 
detailing  the  final  values  from  the  previous  year,  year-to- 
date,  and  current  month  (Table  2).  The  values  are 
available  in  dollars  and  as  a  percentage  of  the  quality 
control  budget.  Being  a  monthly  report,  the  cost  of  quality 
(COQ)  information  is  primarily  used  as  a  tool  for  keeping 
upper  management  aware  of  cost  data. 

In  addition  to  the  division-wide  data  presented  in 
Figure  2  and  Table  2,  prog  ram- spe c i f i c  data  are  also 
available  for  selected  categories  by  month  (Table  3)  and 
year-to-date  . 


Table  2.  Quality  Costs,  YTD  and  Current  Month 


CATIG01!  TUB  HD  TltS  TO  DAT!  COHHT  BOITB 

1198!’  ACTUAL  t  OF  ACTOAL  XOT 

)  iC  • *000  51  ACTOAL  JC  ($000  Si  ICTOil  5C 

PEIMITIOI 

ISS38AICI  IRGiRKIBING 
wOiLITT  A0S1N  SKB?i:iS 

minis a lorifirroi 

tom  PIITIITIOI 
tPPBilSiL 

IICOBIIG  IISPICTIOI 
ILL  OTHIB  IISPICTIOI 
TIST 

STAIDAIDS  A  CALI  BEAT  I  OR 
SOPPLIIB  50ALITT 

mm  iipiisi 

SAI  LS.  A  ITALOATIOI  LIB 

PIODOCT  ACCIPTAHCI 

BICBOILIC  QOALITT  IISP 

TOTAL  APPIIISAL 

FAILOII 

IHOII  Li  BO  I 

ICI/ICI  LABOB 

SCI1? 

WHITT 

TOTAL  FAILOBI 

TOTAL  QQAIITT  COSTS  1)000) 

SALIS  0009) 

9UALITT  COSTS  (TOTAL) 

AS  A  I  0?  SALIS 
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Table  3.  Monthly  Failure  Costs 
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Graphs  of  quality  cost  data  are  presented  in  numerous 


formats  throughout  the  monthly  report:  rework  costs  by 
program  (Figure  3),  rework  as  a  percentage  of  direct  labor 
(Figure  4),  rework  costs  by  function  or  cause  (Figure  5), 
rework  by  function  as  a  percentage  of  function  direct  labor 
dollars  (Figure  6),  engineering  change  request  (ECR)/ 
engineering  change  notice  (ECN)  costs  by  program  (Figure  7), 
ECR/ECN  costs  as  a  percentage  of  direct  labor  dollars 
(Figure  8),  ECR/ECN  costs  by  function  (Figure  9),  ECR/ECN  by 
function  as  a  percentage  of  direct  labor  dollars  (Figure 
10),  and  scrap  costs  by  program  (Figure  11). 


Figure  3.  Rework  Costs  by  Program 


37 


Figure  4.  Rework  as  a  Percentage  of  Direct  Labor  Costs 


PERCENTAGE  IXI 
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Figure  10.  ECR/ECN  Costs  by  Function  as  a  Percentage 
of  Direct  Labor  Dollars 


Figure  11.  Scrap  Costs  by  Program 
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The  monthly  report  concludes  by  identifying  the  major 
contributors  to  ECR/ECN,  rework,  and  scrap  costs  (Table  4). 


Table  4.  Major  Contributors  to  Costs 


PBOGiAi 

icm:« 

ilXCSl 

SCIAP 

:  asst/ 

ASST/ 

,  ASST/ 

Fill  10 

DOLUiS 

FAIT  SO.  DOLUiS 

: PAST  80  DOLLAiS 

AL9-138 

26238’5 

919985 

2610877 

wain 

2628210 

1 2(88960 

262(866 

■3130990 

2625202 

1 2(88782 

2627127 

:262821( 

2628201 

;2(87(2( 

2(83902 

12622516 

2626747 

1 2 1 3  8  7  7 1 

262(302 

13130725 

26262(6 

;S385 10/3 150(3(8 

26282(8 

1262(381 

262!20( 

2612716 

2630179 

,2625267 

TOTAL  DGLLA8S 

A  discussion  of  Figure  5  will  show  how  these  graphs 
could  be  used  by  the  QAS  in  monitoring  the  company's  control 
of  the  program.  Test  and  assembly  are  evident  as  the 
dominant  causes  of  rework.  Although  the  total  rework  costs 
have  declined  in  February,  the  proportion  of  rework  due  to 
test  and  assembly  has  actually  risen.  This  area  should  merit 
the  majority  of  the  QAS ' s  time  when  discussing  rework  costs 
with  the  contractor  and  determining  the  causes  of  the  rework 
requi red . 

Observations ■  Inspection  of  product  can  be  subjective, 
due  to  one  person  seeing  what  another  doesn't  see,  even 
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under  identical  conditions.  This  grey  area  between 
acceptable  and  unacceptable  makes  defining  quality 
difficult,  a  situation  which  can  be  illustrated  clearly  in 
the  area  of  soldering.  Even  the  titles  of  potential  solder 
defects  suggest  ambiguity:  solder  not  smooth  and  shiny, 
insufficient  solder,  excessive  solder  (lead  di see rnabl e ) , 
excessive  solder  (lead  not  di see rnable ) ,  and  poor  wetting. 
Consequently,  the  number  of  defects  attributed  to  a  given 
product  may  depend  on  the  person  counting  the  defects, 
regardless  of  the  experience  level  of  that  person. 

Cost  savings  in  quality  assurance  are  often  difficult 
to  evaluate,  especially  in  the  Return-on-Investment  terms 
usually  used  in  business.  Improvements  in  a  process,  such 
as  computerizing  data  collection  to  allow  more  timely  trend 
detection,  must  be  shared  by  both  quality  and  manufacturing, 
making  the  contribution  by  quality  difficult  to  pinpoint. 
Perhaps  the  easiest,  but  h  ast  practical,  way  of  determining 
the  savings  due  to  quality  improvements  and  enhancements 
would  be  to  remove  or  "turn  off"  those  changes.  Long-term 
removal  of  an  established  computerized  system  would  not 
pinpoint  the  savings  due  to  the  enhancement,  but  would 
instead  reduce  productivity  below  the  pre-enhancement  level. 

It  is  possible  to  show  the  aggregate  effect  of  quality 
programs  over  time  by  observing  an  increase  in  the  quality 
of  the  final  product.  Trying  to  measure  the  effects  of  a 
single  "intangible"  program  such  as  a  quality  improvement 
program  or  a  RMS  design  effort  is  more  difficult.  This 
difficulty  hampers  the  effort  to  develop  indicators  on  the 
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effectiveness  of  intangible  programs.  Tangible  programs, 
such  as  raising  yields  or  lowering  defect  rates,  constitute 
the  majority  of  the  measures  identified  in  this  research. 

Texas  Instruments,  Radar  Systems  Division  (TI/RSD)  ( 3 ) 

Background .  TI/RSD  is  a  division  of  Avionics  Systems, 
which  in  turn  belongs  to  the  TI  Defense  Systems  & 

Electronics  Group.  Located  in  McKinney,  Texas,  there  are 
three  major  Product  Customer  Centers  ( PCCs )  at  TI/RSD: 

LANTI RN/MMR ,  F-lll,  and  NAV/ATTACK .  The  manager 
interviewed  at  TI/RSD  was  responsible  for  the  F-lll  PCC; 
consequently,  the  information  provided  during  this  interview 
applies  primarily  to  the  F-lll  product  line.  Although  some 
of  the  techniques  and  quality  measures  may  apply  to  the 
other  product  lines  as  well,  the  product  lines  are  separate 
and  independently  managed,  and  the  responsible  quality 
assurance  personnel  determine  the  most  appropriate 
indicators  for  use  in  their  programs  (see  Observations  below 
for  further  discussion  on  this  point). 

TI/RSD  espouses  a  dual  quality  philosophy  including 
both  the  satisfaction  of  contractual  requirements  and  the 
satisfaction  of  the  customer,  both  the  final  customer  and 
the  next  process.  To  supplement  contractual  requirements, 
TI/RSD  uses  requirements  commonality  within  work  areas  such 
as  electrostatic  sensitive  device  ( ESD )  handling  and 
soldering.  Requirements  commonality  dictates  that  all 
processes  within  an  area  conform  to  the  same  requirements, 
based  on  the  most  stringent  contractual  requirements  in  that 


area.  For  example,  in  a  room  where  ESD  integrated  circuits 
are  handled,  even  non-ESD  integrated  circuits  will  be 
handled  as  if  they  required  ESD  precautions.  Although  a 
quality  extreme,  this  philosophy  simplifies  the  detection  of 
handling  errors  and  reduces  the  chance  of  ESD  integrated 
circuits  from  being  damaged  due  to  negligence. 

Added  emphasis  on  doing  the  job  right  the  first  time  is 
given  by  TI/RSD's  use  of  the  Material  Review  Board  ( MRB ) . 

The  MRB  is  typically  used  to  disposition  a  nonconforming 
item  into  one  of  several  categories:  rework,  standard 
repair,  return  to  vendor,  scrap,  use-as-is,  or  non-standard 
repair.  Since  the  latter  two  dispositions  require 
government  approval,  TI/RSD  has  eliminated  them  as  options, 
thereby  emphasizing  conformance  to  requirements. 

Project  52  is  a  prgram  emphasizing  regular  weekly 
deliveries  rather  than  making  all  shipments  at  the  end  of 
the  month.  The  Project  has  helped  reduce  errors  by 
avoiding  the  tendency  to  rush  a  lot  of  work  through  in  a 
month-end  rush  to  ship  finished  prcJuct. 

Quality  Indicators. 

Inspection  and  Test  Yields.  A  generalized  yield 
chart  (Table  5)  is  used  regularly  by  the  Quality  and 
Reliability  Assurance  ( QRA )  analyst,  the  individual 
responsible  for  monitoring  the  daily  quality  of  a  Product 
Customer  Center.  This  chart  identifies  the  parts  by  number 
and  by  the  printed  wiring  board  ( PWB )  on  which  they  are 
used,  detailing  the  quantity  of  items  having  passed  through 
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Table  5.  Generalized  Yield  Chart 
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a  given  inspection  or-  test  (see  list  below),  the  yield,  and 

the  SPC  attribute  values  p-bar  and  lower  control  limit. 

This  information  is  further  tabulated  into  total  yields  for 

each  part  number  and  average  values  for  each  inspection  or 

test.  The  yield  information  is  included  for  the  following: 

PWB-level  inspection  after  the  PWB  has  been  component 
stuffed 

PWB-level  inspection  after  the  PWB  has  been  flow 
soldered 

PWB-level  circuit  test  after  flow  solder 
PWB-level  inspection  of  manual  assembly  operations 
PWB  electrical  test  before  thermal  shock 
PWB  electrical  test  after  thermal  shock 
PWB-level  inspection  after  the  PWB  has  been 
encapsulated 

LRU  (Line  Replaceable  Unit)  assembly  electrical  test 
before  environmental  stress  screening 
LRU-level  environmental  stress  screening 
LRU  assembly  inspection  before  acceptance  test 
LRU  assembly  acceptance  test 


Table  5  is  an  abbreviated  version  of  the  chart  actually  used 
by  the  Quality  and  Reliability  Assurance  (QRA)  analyst, 
using  "BQC"  and  " FQC"  as  identifiers  for  the  first  two 
categories  identified  above. 

This  indicator  is  used  to  spot  areas  where  additional 
attention  must  be  focused.  Part  numbers  with  a  low  yield 
can  be  identified  rapidly,  as  can  those  parts  with  p-bar 
values  lower  than  the  average.  See  the  Analysis  (Chapter  V) 
for  a  caution  when  vising  this  indicator. 

First-pass  yields  are  identified  by  LRU,  detailing  the 
trend  over  a  year.  Figure  12  shows  the  percent  yield  for 
acceptance  test,  with  other  categories  including 

LRU-level 

Unit  test  before  burn-in 

Burn-in  (environmental  stress  screening) 

Unit  test  after  burn-in 

Final  quality  control  before  acceptance  test 
Quality  control  configuration  inspection 

PWB-level 

Board-level  inspection 
Flow  solder  quality  control 
PWB  circuit  test 

Electrical  assembly  quality  control 
PWB  unit  test  before  thermal  shock 
PWB  unit  test  after  thermal  shock 
Encapsulation  quality  control 

Defects .  Line  Replaceable  Unit  (LRU)  assembly- 
level  defects  are  accumulated  monthly  and  presented 
graphed  to  identify  defect  quantities  by  LRU  sequence  number 
for  each  LRU.  Figure  13  shows  such  a  graph  for  test 
defects,  with  similar  graphs  prepared  for  inspection  defects 
and  standard  repair  procedure  (SRP)  usage.  Since  the  graphs 
represent  LRU  assembly-level  information,  the  defect  count 
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includes  defects  at  the  printed  wiring  board  and  shop 
replaceable  unit  levels  as  well.  The  standard  repair 
procedure  version  of  this  graph  does  not  actually  identify  a 
defect  quantity ,  but  instead  identifies  the  numbe  r  of  times 
SRPs  were  used  in  correcting  defects. 

Specific  areas  of  interest  are  tracked  separately  by 
TI/RSD.  The  solder  area,  for  example,  plots  component  lead 
solder  defects  weekly  (Figure  14),  as  well  as  flow  solder 
defects,  and  touch-up  solder  defects. 


Figure  14.  Component  Lead  Solder  Defects 


TI/RSD  also  uses  what  they  refer  to  as  "pedigree 
graphs"  to  identify  the  long-term  defect  history  of  a  given 
LRU  (including  all  defects  due  to  paperwork,  documentation, 
workmanship,  test  procedure,  planning,  work  instructions, 
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design,  configuration,  handling,  and  cosmetics).  The 
pedigree  data  is  accumulated  from  the  monthly  data  described 
above  and  represented  as  either  a  discrete-point  graph 
(Figure  15)  or  as  a  25  unit  cumulative  average  graph  (Figure 
16).  Both  of  these  graphs  are  useful  for  displaying  defect 
trends  to  upper  management.  The  pedigree  graphs  are  also 
available  showing  only  inspection  defects  or  only  test 
defects  . 

Pareto  analysis  is  used  to  identify  those  areas 
requiring  the  most  immediate  attention.  Figure  17  is  an 
example  of  a  graph  detailing  the  PWBs  and  SRUs  with  the  most 
defects  per  unit,  while  Figure  18  represents  similar 
information  focusing  on  the  defects  occurring  most  often. 


Figure  15.  Discrete  Point  Pedigree 
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Figure  18.  Pareto  Analysis  of  Defect  Occurance 


Engineering  Change  Notices.  ECNs  provide  one 
measure  of  the  number  of  changes  associated  with  a  product. 
As  indicated  by  Figure  19,  ECNs  can  be  due  to  documentation 
errors,  design  errors,  or  improvement  efforts.  This 
information  is  accumulated  at  the  division  level,  and  is 
also  presented  in  a  tabular  format  (Table  6).  The  table 
identifies  the  areas  of  responsibility  for  ECNs,  as  well  as 
the  reasons  for  the  ECNs. 

Customer  Return  Material.  TI/RSD  tracks  the  flow 
of  product  returned  to  them  for  action  (generally  for 
repair).  Figure  20  identifies  the  various  stages  of  flow 
that  customer  return  material  (CRM)  follows,  and  the  number 
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Figure  20.  Stages  of  Customer  Return  Material  Flow 


of  units  a  Product  Customer  Center  ( PCC )  has  at  each  of 
those  stages  on  a  given  date.  The  stages  are  defined 
as : 

MDR  -  Material  Deficiency  Report 
INWORK  -  Unit  in  work 
AWP  -  Awaiting  parts 
RTV  -  Return  to  vendor 
ENGHLD  -  Engineering  hold 

ATC/TI  -  Acceptance  test  complete/waiting  TI  action 
ATC/CUST  -  Acceptance  test  complete/waiting  customer 
action 

PACK  -  In  packaging  for  shipment 
Figure  21  details  the  length  of  time  units  have  been  in  the 
CRM  flow  (used  to  identify  bottlenecks),  and  Figure  22  shows 
the  turnover  and  balance  of  CRM  over  the  past  year. 

Observations.  The  quality  indicators  used  on  a  given 
program  change  as  the  program  itself  matures  from  the  early 
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Figure  21.  Length  of  Time  in  the  CRM  Flow 


Figure  22.  CRM  Turnover  and  Balance 
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months  of  production  to  the  final  months  of  the  program.  As 
the  F-lll  program  has  matured,  for  example,  TI/RSD  has 
shifted  focus  from  test  rework  to  first  pass  yield  to 
defects  per  unit.  This  requirement  for  having  dynamic 
quality  assurance  indicators  can  present  a  contractual 
problem:  when  a  contract  is  definitized  several  months 

(often  over  a  year)  before  production  under  that  contract  is 
started,  it  is  impossible  to  identify  the  single  set  of 
indicators  which  would  be  most  appropriate  for  the  duration 
of  that  production  effort. 

Electronic  Systems  Division  (ESP)  (25) 

Background .  ESD  is  a  division  of  Air  Force  Systems 
Command,  the  agency  responsible  for  acquiring  weapon  systems 
for  the  Air  Force.  ESD  is  responsible  for  developing  and 
acquiring  systems  such  as:  long-range,  ground-based  radars; 
satellite  communications  terminals;  the  Joint  Tactical 
Information  Distribution  System  (JTIDS);  the  Joint 
Surveillance  Target  Attack  Radar  System  (JSTARS);  and  the 
Airborne  Warning  and  Control  System  (AWACS). 

ESD  defines  quality  as  meeting  all  contractual 
requirements.  With  this  definition,  ESD  does  not  limit  its 
quality  surveillance  solely  to  the  requirements  of 
MIL-Q-9858A,  but  instead  charges  its  personnel  with 
monitoring  all  aspects  of  the  contract  with  emphasis  on  the 
quality  and  manufacturing  issues.  Several  of  the 
indicators,  refered  to  as  metrics  by  ESD,  will  also  be  of 
interest  to  the  manufacturing  community,  and  communication 
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will  help  prevent  duplication  of  effort.  Total  Quality 
Management  (TQM)  fits  in  well  with  the  ESD  philosophy. 

Quality  Indicators. 

Pareto  Analysis.  ESD  uses  Pareto  analysis 
extensively  for  tracking  quality  problems  and  progiess 
towards  resolving  those  problems  (see  Figures  17  and  18  for 
examples  of  Pareto  analysis).  Since  the  principle  behind 
Pareto  charts  is  that  the  majority  of  problems  are  the 
result  of  a  few  problems,  this  indicator  lends  itself  well 
to  tracking  various  defect  and  yield  problems. 

Drawings  Released.  The  evolving  design  progress 
is  tracked  to  indicate  the  gap  (if  any)  between  the  design 
objective  and  manufacturing  technology  and  pr oduc ibi 1 i ty , 
highlighting  any  potential  schedule  problems.  Figure  23 
shows  the  actual  numbers  of  drawings  released  compared  to 
the  number  scheduled  for  release,  while  Figure  24  compares 
percent  actually  released  to  percent  scheduled  for  release. 
Drawings  are  tracked  by  quality  personnel  since  they  must  be 
approved  by  quality  prior  to  release  (supporting  the  TQM 
philosophy  of  ESD).  Since  drawings  must  be  completed  before 
final  manufacturing  plans  and  work  instructions  can  be 
completed,  this  indicator  can  reveal  early  signs  of 
potential  problems. 

Work  Instructions.  Work  instructions  are  tracked 
similarly  to  drawing  releases  (i.e.,  by  actual  numbers  and 
by  percentages).  Quality  is  involved  due  to  the  requirement 
for  quality  personnel  to  determine  the  appropriate 
inspection  points. 
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Figure  23.  Drawings  Released  vs.  Scheduled 


Figure  ^4.  Percent  of  Drawings  Released  vs.  Scheduled 
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Purchase  Orders  (POs).  Purchase  orders  also 


require  input  from  quality  assurance  and  are  tracked  to 
assure  all  subcontracted  items  are  ordered  in  a  timely 
manner.  POs  are  tracked  using  formats  similar  to  Figures  23 
and  24.  This  information  verifies  that  the  scheduled  number 
of  POs  have  been  issued,  but  does  not  verify  that  the 
correct  items  have  been  ordered. 

Material  Review  Board  (MRB).  Tracking  MRB 
activity  can  help  identify  poor  hardware  producibi 1 i ty , 
process  deterioration,  personnel  training  needs,  and  misuse 
of  the  minor  waiver  process.  As  noted  previously,  MRB 
disposition  categories  include  scrap,  rework,  repair,  return 
to  vendor,  and  use  as  is.  Excessive  dispositions  in  any  of 
these  categories  can  identify  management  need  to  control 
nonconformances.  Figure  25  identifies  the  actual  numbers  of 
MRB  actions  within  the  facility  and  in  the  program  of 
interest,  while  Figure  26  compares  the  percentage  of  MRB 
actions  by  the  program  to  the  total  number  of  actions. 

Scrap,  Rework,  Repair  ( 5RR ) .  Scrap  is  usually 
defined  as  the  cost  of  material  and  value-added  labor 
invested  in  an  item  no  longer  usable  by  the  program.  Rework 
is  additional  value-adder  labor  required  to  bring  an  item  in 
conformance  with  specification  (often  the  result  of  work 
instruction  steps  not  accomplished).  Repair  is  additional 
value-added  labor  required  to  bring  an  item  to  a  condition 
where  it  can  be  used  as  is  (e.g.,  plugging  a  misdrilled  hole 
and  redrilling  the  hole  in  the  correct  location).  Tracking 
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the  amount  of  SRR  within  a  program  can  identify  process 
control  problems,  producibi 1 i ty  problems,  and  the  need  for 
modernization  and  training.  Low  SRR  rates  indicate 
effective  quality  planning  by  way  of  a  design  that  is  free 
from  error  and  one  which  can  be  easily  produced  and 
controlled.  ESD  analyzes  SRR  rates  as  a  percentage  of 
production  costs,  defined  as  labor  costs.  Figure  27  shows 
actual  SRR  costs  and  production  costs,  while  Figure  28  shows 
SRR  as  a  percentage  of  production  costs. 

Yield .  ESD  monitors  yield  rates  to  reveal 
information  on  process  control,  workmanship,  the  need  to 
improve  pr oduc i bi 1 i ty ,  and  the  need  for  manufacturing 
technology  improvements.  Screening  at  lower  assembly  levels 


150.80-, 

Prod  Cosh 

m 

100,00- 

b_b_b-b-b-c  -  Costs 

]-G"B 

L 

n 

50.00- 

A 

n 

Cj 

0,00  - 

-  ^  -&  'A— £— ^---a — A — A — A — A 

1  1  l  1  !  1  1  1  1  1  1 

on  Mar  Hay  Jyl  Sep  Nov 

Figure  27.  Actual  SRR  and  Production  Costs 


61 


is  suggested  as  a  method  for  improving  the  yield  at 
subsequent  inspections  and  tests.  While  ESD  acknowledges 
the  likelihood  of  low  yields  for  new  or  special  process, 
quality  personnel  expect  improving  trends  with  process 
maturity.  Yields  of  100%  are  the  desired  goal  where 
feasible.  A  general  format  for  yield  charts  was  shown  in 
Figure  12 . 

Observations.  Perfection  is  a  utopia  not  likely  to  be 
achieved  during  a  program.  Because  of  this,  mutually  agreed 
to  goals  must  be  imposed  on  appropriate  portions  of  the 
quality  program.  These  goals  should  be  based  on  past 
performance,  yet  should  emphasize  continuous  improvement. 

The  QAS  must  realize,  however,  that  additional  improvement 
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may  not  be  possible  at  a  given  level  of  technology.  When 
technology  becomes  the  limiting  factor,  the  technology 
itself  must  be  improved  to  continue  the  improvement  process. 

A  good  quality  system  is  necessary,  but  is  not  enough 
to  insure  a  high  quality  product:  a  contractor  with 
seemingly  a  good  quality  system  may  still  produce  a  poor 
product.  Likewise,  evaluating  a  contractor's  quality  system 
is  not  enough  to  insure  product  quality.  Visible  management 
commitment  is  also  a  necessary  ingrediant. 

Aeronautical  Systems  Division  (ASP)  ( 2 ) 

Background .  ASD  is  a  division  of  Air  Force  Systems 
Command,  the  agency  responsible  for  acquiring  weapon 
systems  for  the  Air  Force.  ASD  is  responsible  for 
developing  and  acquiring  systems  such  as:  the  Bl-B;  the 
Advanced  Technology  Bomber  ( B  —  2  )  ;  the  Advanced  Tactical 
Fighter;  the  Short  Range  Attack  Missile  II  (SRAM  II);  the 
F-15  Eagle;  the  F-16  Fighting  Falcon;  and  the  Maverick 
Missile . 

ASD  defines  quality  as  conformance  to  contractual 
requirements,  with  the  understanding  that  fitness  for  use 
is  a  nr  "or  consideration  when  making  contract-related 
decisions.  With  this  philosophy,  the  decision  on  whether 
to  grant  a  waiver  or  deviation  is  based  on  the  usability  of 
the  product  in  the  field,  and  not  solely  on  whether  or  not 
the  product  meets  the  contractual  requirement.  This 
acceptability  criteria  allows  flexibility  in  the  decision 
making  process. 
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Indicators .  The  ASD  managers  did  not  present  any 


particular  formats  that  they  use  to  gather  information. 

Instead,  they  stressed  the  necessity  of  following  the 

intent  of  MIL-Q-9858A  over  the  more  specific  guidance 

offered  in  Handbook  H50.  The  focus  of  an  evaluation  should 

be  as  stated  in  the  summary  section  of  MIL-Q-9858A  (Quality 

Program  Requirements),  that  the  contractor  has 

An  effective  and  economical  quality  program,  planned 
and  developed  in  consonance  with  the  contractor's 
other  administrative  and  technical  programs  .... 
(6:1) 

An  evaluation  based  on  this  focus  would  look  at  each  process 
and  evaluate  if  it  is  being  done  correctly,  with  breadth  of 
coverage  being  a  major  consideration.  Each  functional 
group,  including  engineering,  configuration,  test, 
contracts,  shipping,  and  receiving,  should  have  measurable 
indicators  that  identify  how  well  they  are  meeting  the 
needs  of  the  next  process. 

In  addition  to  evaluating  the  indicators  of  each 
functional  organization,  the  QAS  must  identify  the 
effectiveness  of  those  indicators.  The  indicators  must  be 
reviewed  and  acted  upon  in  a  timely  manner,  not  merely 
passed  up  the  coordination  chain.  Just  as  Deming 
disapproves  of  posters  in  the  workplace  taking  the  place  of 
true  management  commitment,  indicators  can  not  be  allowed 
to  become  paperwork  exercises  fulfilling  policy 
requirements.  By  nature,  indicators  are  only  identifiers 
of  problems;  the  investigation  of  causes  and  elimination  of 


64 


those  causes  is  still  the  major  goal  of  quality  imrovement 
efforts . 

Indicators  must  be  used  to  identify  as  many  of  the 
needs  of  the  customer  as  practical.  An  indicator  measuring 
only  the  turnaround  time  of  a  contract  modification  does 
not  address  what  may  be  the  more  important  issues  of 
accuracy  and  completeness;  a  modification  with  a  rapid 
turnaround  that  is  returned  because  of  errors  does  not  meet 
the  needs  of  the  customer.  Consequently,  the  QAS  must  be 
prepared  to  not  only  determine  the  breadth  of  functional 
coverage  and  the  effectiveness  of  the  indicators  used,  but 
must  also  determine  the  breadth  of  coverage  within  the 
function . 

ASD  managers  acknowledge  the  usefulness  of  the 
indicators  being  used  at  the  other  locations  studied.  in 
particular,  cost  of  qualtiy  ( COQ )  information  was  noted  to 
be  useful  when  attempting  to  convince  a  contractor  of  the 
need  for  quality  initiatives  in  his  company.  When  a 
manager  realizes  what  portion  of  his  revenue  is  lost  to 
defects,  failures,  and  inspections,  he  will  be  more  inclined 
to  implement  preventative  quality  efforts.  COQ  information 
is  useful  both  as  an  attention  getting  tool  and  as  a  means 
of  demonstrating  how  focusing  the  quality  effort  towards 
defect  prevention  can  be  rewarding. 

Obse  rvations .  Beyond  the  traditional  definitions,  ASD 
managers  view  quality  as  a  management  application  of  the 
quality  discipline.  Supporting  the  Total  Quality 
Management  (TQM)  ideas,  this  view  recognizes  that  quality 
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responsibility  does  not  reside  solely  in  the  Quality 
Assurance  Department,  but  is  instead  a  universal  management 
concern.  The  quality  department  should  act  as  a  catalyst  to 
the  quality  efforts  being  achieved  throughout  the  company, 
assisting  with  the  technical  details  of  the  quality  effort 
as  needed  and  providing  encouragement  to  continue  when  the 
results  of  the  efforts  are  slow  to  materialize. 

The  concept  of  viewing  the  next  process  as  customer  is 
also  espoused  by  ASD  managers.  This  idea  requires 
communication  among  the  functional  departments  and  the 
program  office,  each  group  defining  what  inputs  they  require 
to  do  their  job  (providing  another  group  with  some  sort  of 
input)  correctly  and  efficiently.  Considering  the  next 
process  as  customer  enables  each  function  to  define  goals 
and  see  how  well  those  goals  are  being  attained.  This  idea 
also  supports  the  fitness  for  use  definition  given 
earlier — the  needs  of  the  next  process  should  guide  the 
requirements  being  enforced. 

A  quality  improvement  effort  is  intended  to  reduce 
variation  in  the  normal  process,  allowing  more  of  the  work 
passing  through  to  be  within  acceptable  standards.  Through 
quality  improvement,  the  contractor  can  ultimately  cut  costs 
for  both  himself  and  the  government.  A  barrier  to  quality 
improvement  programs,  however,  is  the  structure  of  the 
contract:  the  profit  for  a  contractual  effort  is  commonly 

tied  to  the  cost  of  doing  that  effort.  In  reducing  costs  on 
the  current  contract,  the  contractor  will  ultimately  reduce 
costs  on  future  contracts  and  his  profit  on  those  contracts. 


A  system  of  allowing  sufficient  profit  while  encouraging 
cost  saving  techniques  such  as  quality  improvement  efforts 
is  required,  and  experimental  contracting  efforts  are  being 
implemented  to  study  different  applications  of  profit. 
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v.  Results  and  Analysis 


This  chapter  provides  the  results  and  analysis  of  the 
study,  beginning  by  addressing  the  issues  raised  in  the 
first  three  chapters.  The  issues  are  followed  by  a 
discussion  about  the  new  Quality  Assurance  Specialist  ( QAS ) 
and  the  situation  faced  when  first  entering  a  system  program 
office.  Ideas  presented  here  may  assist  the  QAS  in  putting 
his  new  assignment  in  the  correct  perspective.  Following 
this  is  a  presentation  of  quality  program  elements  that 
might  be  found  in  an  "ideal"  company.  The  appropriate 
quality  indicators  are  discussed  here,  along  with  their 
applications,  merits,  and  shortfalls.  Finally,  barriers  to 
the  quality  program  are  covered,  identifying  various  problem 
areas  both  the  QAS  and  the  contractor  may  face  while 
developing  an  appropriate  set  of  quality  assurance 
indicators. 

Issues  Raised  in  Chapters  I  -  III 

MIL-Q-9858A.  The  first  issue  centers  on  whether 
MIL-Q-9858A,  Quality  Program  Requirements,  is  an  outdated 
document  and  should  be  revised,  or  whether  it  still  contains 
valid  guidance.  Discussions  during  this  study  levealed  that 
the  quality  managers  believe  MIL-Q-9858A  is  still  a  valid, 
useful  document.  The  format  of  MIL-Q-9858A  as  general 
guidance  has  allowed  it  to  apply  equally  well  now  as  it  did 
25  years  ago.  MIL-Q-9858A  does  not  attempt  to  support  any 
quality  philosophy  over  another,  but  instead  supports  the 
idea  that  a  good  quality  program  is  a  necessary  ingredient 


ingredient  in  quality  product.  By  not  requiring  the 
development  and  philosophies  of  that  quality  program  to  be 
identical  for  each  company,  MIL-Q-9858A  has  allowed  the 
companies  to  tailor  programs  to  meet  requirements. 

MIL-Q-9858A  does,  however,  require  certain  central 
elements  to  be  present  in  the  quality  program.  These 
elements  are  not  intended  to  limit  the  contractor's  program, 
but  instead  to  provide  a  foundation  on  which  the 
contractor's  program  can  be  developed.  As  discussed  in 
Chapter  II,  additional  reading  in  the  Department  of  Defense 
publications  governing  quality  programs  reveals  the 
expectation  of  quality  programs  to  support  both  the 
conformance  to  requirements  definition  of  quality  as  well  as 
the  fitness  for  use  definition.  Information  obtained  during 
this  study  supports  the  contractor's  intent  to  comply  with 
this  expectation.  They  have  developed  their  programs 
starting  with  the  central  elements  and  expanded  them  to 
include  the  fitness  for  use  philosophy.  Emphasis  on 
fitness  for  use  is  evident  in  Texas  Instrument's  (TI) 
elimination  of  the  need  for  government-approved  material 
review  board  decisions,  ITT's  progress  in  computer-aided 
design,  and  the  overall  support  of  "next  process  as 
custome  r . " 

Although  MIL-Q-9858A  is  not  considered  to  be  outdated, 
the  need  for  more  specific  guidance  is  recognized.  Both  ITT 
and  TI  have  established  policies  based  on  their 
interpretation  of  MIL-Q-9858A.  These  policies  guide 
everyone  from  management  to  the  floor  workers  in  how  their 
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tasks  should  be  performed  to  comply  with  requirements, 
identifying  what  to  do,  how  to  do  it,  and  how  often  to  do 
it.  Although  these  policies  are  flexible  enough  to  change 
with  changing  technology,  they  are  expected  to  be  rigidly 
followed  while  in  place. 

The  government  also  recognizes  the  need  for  specific 
guidance.  Electronic  Systems  Division  (ESD;  quality 
managers  have  developed  a  collection  of  metrics  to  be  used 
when  evaluating  contractors,  including  both  manufacturing 
and  quality  assurance  issues.  Aeronautical  Systems  Division 
(ASD)  quality  managers  are  developing  a  similar  guide  for 
use  at  ASD.  Based  on  the  requirements  and  intent  of 
MIL-Q-9858A,  these  guides  are  intended  to  supplement  the 
Quality  Program  Requirements  guidance. 

Research  Questions . 

Quality  Definition.  How  is  quality  defined  in  the 
industry?  The  contractors  are  paid  according  to  how  well 
they  meet  contract  requirements.  If  they  go  over  the 
expected  cost,  they  lose  some  profit  (in  firm,  fixed-price 
contracts  which  are  typical  for  production  efforts).  They 
may  be  penalized  financially  for  missing  delivery  schedules 
or  for  providing  products  that  work  in  the  field  but  do  not 
meet  specification.  Because  of  this,  the  contractors  are 
required  to  have  a  quality  definition  which  supports  the 
conformance  to  requirements  philosophy.  This  research  has 
found  this  to  be  true.  In  addition,  the  research  has  shown 
that  the  government  also  mentions  conformance  to 
requirements  as  their  primary  philosophy. 
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Despite  the  financial  bias  supporting  conformance  to 
requirements,  the  fitness  for  use  philosophy  also  played  a 
large  part  in  each  of  the  four  location's  overall  quality 
philosophy.  The  fitness  for  use  philosopy  was  most  evident 
in  the  broad-based  support  of  considering  the  next  process 
as  the  customer  and  in  the  concern  for  meeting  minimum 
standards.  As  new  methods  of  determining  contract  profits 
are  developed,  the  fitness  for  use  quality  philosophy  will 
be  allowed  an  even  greater  influence  on  the  quality  program. 

Emphasi s .  Which  areas  of  the  contractor's  quality 
program  should  be  emphasized  in  determining  the  status  of 
the  quality  program?  The  contractors  tended  to  place  more 
emphasis  on  quality  evaluation  of  the  areas  that  were 
subject  to  quantifiable  analysis.  These  tangible  areas  of 
the  programs  were  identified  in  Chapter  IV.  In  this 
research,  neither  contractor  relied  on  a  single  indicator 
more  than  any  other,  realizing  the  need  for  a  comprehensive 
quality  program.  A  given  area  would  receive  added  attention 
if  a  problem  were  to  develop  in  that  area,  but  daily 
attention  was  given  each  area  by  individuals  responsible  for 
the  details  of  that  area. 

The  government,  however,  focused  primarily  on  the  more 
intangible  areas  of  the  contractors'  quality  programs, 
especially  on  the  active,  visible  management  commitment  to 
quality.  No  measures  were  presented  for  objectively 
evaluating  this  commitment,  however.  In  lieu  of  being  able 
to  provide  objective  measures  of  management  commitment,  the 
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government  used  the  same  tangible  indicators  as  the 
contractors . 

Current  Indicators.  What  quality  indicators  are 
currently  being  used  by  both  the  industry  and  the 
government?  The  indicators  outlined  in  Chapter  IV  and 
discussed  below  as  being  used  by  the  "ideal"  quality  program 
are  the  quality  indicators  currently  being  used  by  ITT,  TI , 
ESD ,  and  ASD.  Although  the  individual  indicators  are  not 
used  universally,  all  focus  on  the  tangible  areas  of  the 
quality  program.  Indicators  in  use  either  identify  areas 
where  additional  attention  is  necessary  immediately  to  solve 
a  problem  (e.g.,  yields  and  defects)  or  areas  where 
potential  problems  exist  (e.g.,  statistical  process  control 
( SPC )  and  drawing  release  summaries). 

The  indicators  currently  used  also  focus  on  the 
conformance  to  specification  apects  of  the  quality  program, 
that  is,  they  identify  areas  where  the  product  may  not  meet 
standard.  As  focus  shifts  more  towards  fitness  for  use  or 
customer  satisfaction,  the  indicators  may  shift  more  towards 
monitoring  the  non-hardware  processes  within  the  company. 

Information  Basis  .  What  information  is  used  as  a 
basis  for  each  indicator?  As  indicated  by  the  format 
examples  presented  in  Chapter  IV,  and  as  dictated  by  the 
nature  of  the  tangible  indicators,  the  quality  indicators 
are  based  primarily  on  hardware-process  data,  including  the 
drawings,  vendors,  and  purchase  orders  supporting  the 
production  of  the  hardware. 
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The  specific  basis  for  each  indicator  can  be  found  in 
the  sections  describing  the  indicator. 

Indicator  Usage.  Who  uses  each  indicator,  and 
why?  With  the  exception  of  the  cost  of  quality  ( COQ ) 
indicators  and  the  pedigree  charts,  each  indicator  is  used 
by  the  quality  assurance  manager  responsible  for  monitoring 
and  controlling  the  quality  of  the  production  line.  The 
information  is  accumulated  on  a  daily,  weekly,  or  monthly 
basis  to  identify  either  current  problems  or  potential 
problems.  The  fact  that  an  indicator  is  not  used  on  a  daily 
basis  does  not  negate  its  importance.  Customer-returned 
material,  for  example,  typically  does  not  accumulate  rapidly 
enough  to  make  daily  information  feasible  or  useful.  The 
frequency  of  use  will  depend  on  the  aspect  of  the  quality 
program  the  indicator  is  intended  to  monitor. 

The  COQ  and  pedigree  indicators  are  used  to  keep  upper 
management  abreast  of  the  status  of  two  portions  of  the 
quality  program.  The  time  required  to  obtain  COQ 
information  (often  four  to  six  weeks)  and  the  long-term 
nature  of  the  pedigree  information  make  these  two  measures 
almost  useless  as  shop  floor  management  tools.  They  are 
more  suited  as  tools  upper  management  might  use  to  evaluate 
the  efforts  of  the  quality  managers  themselves. 

Unused  Indicators.  What  indicators  are  not 
currently  being  used  by  the  Air  Force  that  may  provide  a 
better  assessment  of  the  quality  status  of  the  contractor? 
This  research  did  not  identify  any  indicators  used  by  the 
industry  but  not  used  in  at  least  a  sim;lar  format  by  the 
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government.  Since  the  contractor  interviews  were  both 
performed  before  the  government  interviews,  the  indicators 
used  by  industry  were  discussed  with  the  government 
managers.  The  government  managers  were  familiar  with  each 
of  the  indicators  used  by  the  contractors,  but  did  not 
necessarily  monitor  each  of  those  areas  on  a  frequent  basis. 
Vendor  ratings  for  subcontractors,  for  example,  were  not 
evaluated  regularly  unless  a  problem  arose  in  that  area. 

The  contractor,  however,  felt  it  necessary  to  monitor  the 
quality  of  his  incoming  material  frequently.  The  fact  that 
the  government  managers  might  not  be  using  an  indicator  on  a 
regular  basis  was  not  identified  by  either  side  to  be  a 
deficiency  in  government  monitoring. 

The  New  Quality  Assurance  Specialist  ( QAS ) 

The  newly  assigned  QAS  is  at  a  disadvantage  upon  first 
entering  the  system  program  office:  his  peers  are  more 
experienced  and  his  supervisor  expects  a  rapid  learning 
curve.  The  first  part  of  this  section  will  offer  a  way  to 
deal  with  the  problems  of  the  learning  curve,  and  explain  a 
use  for  the  information  provided  by  this  study.  Following 
will  be  a  discussion  on  what  the  focus  of  quality  indicators 
should  be.  Finally,  before  describing  elements  of  an 
"ideal"  quality  program,  the  need  for  flexibility  in  the 
QAS ' s  use  of  quality  indicators  will  be  addressed. 

QAS  Education  and  Training.  The  new  Quality  Assurance 
Specialist  (QAS)  ’-as  guidance  available  from  superiors, 
peers,  organizational  operating  instructions,  MIL-Q-985CA, 


and  Handbook  H50,  along  with  the  countless  publications  on 
quality,  manufacturing,  engineering,  configuration, 
contracts,  and  other  functional  organizations.  These 
sources  are  not  all  published  by  the  Department  of  Defense 
and  are  available  from  the  Government  Printing  Office, 
libraries,  and  bookstores.  From  these  abundant  sources  of 
knowledge,  however,  the  QAS  must  be  able  to  distill  the 
information  most  useful  in  accomplishing  the  new  job. 

Compounding  the  problem  of  overwhelming  information  is 
the  expectation  that  the  QAS  can  "pick  up"  the  knowledge  on 
the  job.  Rarely  is  the  QAS  given  any  formal  education  in 
the  general  area  of  quality  assurance  which  might  simplify 
the  search  for  appropriate  guidance.  Instead,  the  QAS  is 
given  on-the-job  training,  typically  under  the  "fire 
fighting"  atmosphere  of  daily  systems  program  office 
activity.  This  type  of  education  focuses  more  on  problem 
solving  than  on  problem  prevention.  This  initial  experience 
can  bias  the  QAS  towards  expecting  quality  programs  to 
emphasize  defect  detection  more  than  prevention. 

Insufficient  formal  education  coupled  with  biased  on- 
the-job  training  results  in  QASs  who  are  uncertain  about  the 
nature  of  their  job.  Given  sufficient  time,  the  QAS  will 
receive  the  necessary  formal  education  and  garner 
information  from  the  various  publications.  How  more 

time  than  should  be  necessary  will  be  spent  in  cms 
education  process,  during  which  the  QAS  will  be  performing 
below  his  or  her  potential. 


Understanding  that  the  current  on-the-job  training  of 
QASs  is  not  the  ideal  situation,  this  study  attempts  to 
provide  a  basic  review  of  certain  quality  assurance 
indicators  currently  used  in  the  defense  electronics 
industry.  The  reader  will  be  able  to  review  the  indicators 
and  observations  presented  in  this  study  for  applicability 
to  his  program.  The  indicators  are  being  used  by  defense 
electronics  companies  with  excellent  quality  programs  and  by 
the  government  offices  monitoring  those  contractors.  The 
indicators  presented  should  provide  a  foundation  while  not 
limiting  innovative  quality  progress  by  either  the  QAS  or 
the  contractor. 

QAS  Focus.  This  study  focused  primarily  on  tangible 
quality  assurance  indicators,  due  more  to  the  relative  ease 
of  maintaining  these  indicators  than  to  the  relative 
importance  of  tangible  indicators  over  "intangible" 
indicators.  In  fact,  the  QAS  should  focus  on  the  entire 
quality  program,  not  just  the  hardware  relate.,  areas.  The 
Total  Quality  Management  philosophy  instilled  in  the  quality 
programs  of  each  location  studied  supports  quality  being  a 
part  of  each  individual  and  each  process,  and  ESD  reminds 
the  reader  that  a  good  quality  system  alone  does  not 
guarantee  a  quality  product.  ASD  concurs  with  this  view  by 
suggesting  that  the  quality  program  meet  three  goals: 
breadth  of  quality  commitment  throughout  the  company, 
breadth  of  quality  commitment  within  each  organization,  and 
effectiveness  of  the  quality  commitment  within  e?rh 
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organization.  Posters,  slogans,  and  indicators  are  not  a 
substitute  for  quality  commitment. 

The  QAS  must  realize  that  even  within  the  same  company 
the  typical  product  line  is  independent  in  many  respects 
from  the  other  product  lines.  Because  of  this  separation, 
the  types  of  indicators  used  and  the  quality  philosophy 
followed  may  vary  from  one  product  line  to  the  next.  The 
differences  in  quality  management  are  likely  to  be 
considerably  greater  between  companies.  Differing 
philosophies  and  varying  indicators  make  comparisons  with 
other  companies  or  product  lines  virtually  impossible. 
Focusing  on  a  single  program  and  the  improvements  that  can 
be  made  to  the  quality  efforts  of  that  program  helps  foster 
an  attitude  of  cooperation  from  the  contractor. 

Flexibility.  The  QAS  should  review  the  quality 
indicators  and  observations  presented  here  in  light  of  his 
quality  program.  Some  indicators  may  be  applicable,  others 
not.  As  indicated  by  Texas  Instruments,  the  QAS  must  be 
flexible  when  choosing  quality  indicators  to  apply  to  a 
program.  As  a  program  matures  from  the  early  stages  of 
production  to  the  last  months  of  the  production  run,  the 
relative  importance  of  the  program's  indicators  will  change 
accordingly.  The  need  for  good  communication  between  the 
contractor  and  the  government  QAS  is  evident.  The 
experience  of  the  contractor  will  prove  invaluable  when 
attempting  to  define  the  indicators  that  will  be  required  by 
a  contract  whose  effort  will  not  begin  for  several  months, 
even  a  year. 
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As  noted  above,  different  companies  and  programs  use  a 
variety  of  quality  indicators.  Although  some  of  these 
differences  are  due  to  the  management  style  of  the 
contractor,  other  differences  may  be  due  to  differing  data 
analysis  capabilities  of  the  programs.  While  one  program 
may  employ  extensive  automation  in  data  collection  and 
analysis,  allowing  rapid  output  of  a  variety  of  reports, 
another  program  may  rely  on  manual  collection  and  analysis 
of  data.  It  would  be  unrealistic  to  expect  the  two 
programs  to  be  equally  receptive  to  innovation.  The  QAS 
must  be  flexible  not  only  to  different  management 
philosophies  and  indicators,  but  also  to  the  physical 
limitations  of  individual  programs. 

The  "Ideal"  Quality  Program 

This  study  has  identified  several  key  areas  of  a 
contractor's  quality  program  based  on  the  excellent  quality 
programs  at  ITT  and  Texas  Instruments  and  supplemented  with 
information  from  ESD  and  ASD.  This  section  identifies  those 
key  elements,  beginning  with  the  quality  management  at  the 
contractor.  Following  this  will  be  a  discussion  of  the 
program  areas  where  quality  indicators  play  a  role,  and  a 
review  of  the  importance  of  quality  indicator  visibility. 
Automation  as  an  aid  to  the  quality  program  will  be  covered 
next,  followed  by  a  discussion  focused  on  continuous 
improvement.  This  section  concludes  with  a  review  of  the 
changing  trend  in  philosophies  from  the  current  idea  of 


78 


conformance  to  requirements  to  the  emerging  considerations 
of  fitness  for  use. 

Quality  Management.  Management  should  be  at  the  heart 
of  the  quality  efforts  within  the  company.  Handbook  H50 
discusses 

An  effective  and  economical  quality  program,  planned 
and  developed  in  consonance  with  the  contractor's  other 
administrative  and  technical  programs  ....  (6:1) 

In  addition,  MIL-Q-9858A  describes  the  quality  program  as 

assuring  ".  .  .  adequate  controls  throughout  all  areas  of 

contract  performance;  e.g.,  development,  manufacturing,  and 

shipping"  (4:3).  Clearly  quality  responsibility  lies  with 

the  company's  management  and  not  solely  with  the  manager  of 

the  quality  assurance  department.  A  program  that  is 

integrated  with  the  other  programs  within  a  company  must  be 

backed  by  a  center  of  authority  and  responsibility;  a 

program  without  management  support  will  falter. 

Although  the  QAS  may  find  it  difficult  to  measure 

actual  management  commitment  and  involvement,  indications 

should  be  evident.  The  quality  assurance  function  should  be 

at  the  same  organization  level  as  manufacturing  and 

engineering,  for  example.  A  quality  function  reporting  to 

the  manufacturing  manager  runs  the  risk  of  being  placed 

second,  behind  manufacturing  and  schedule  concerns.  The 

receptiveness  of  the  other  functional  organizations  to 

quality  initiatives  is  another  indication  that  upper 

management  places  a  priority  on  quality.  The  order  of 

appearance  of  the  quality  function  and  the  time  alloted  to 

their  report  at  the  weekly  and  monthly  program  meetings,  and 
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the  periodic  company  meetings,  is  a  manifestation  of 
management  commitment.  Putting  quality  last  on  the  list  at 
meetings,  when  everyone  is  anxious  to  get  back  to  work,  may 
indicate  quality's  true  priority  at  the  company. 

The  requirement  for  upper  management  commitment  does  not 
preclude  the  necessity  of  having  a  quality  assurance 
department.  This  department  must  be  available  to  aid 
management  in  the  implementation  of  their  commitment. 

Proper  education  and  training  are  required  to  insure  harmony 
of  the  quality  program  with  the  other  programs  at  the 
company,  as  well  as  to  monitor  the  effectiveness  of  the 
quality  program.  Engineering,  configuration,  and  contracts 
management,  not  skilled  in  the  development  of  quality 
indicators,  must  be  aided.  The  quality  department  also 
serves  as  a  focal  point  for  interfacing  with  the  government 
QAS  . 

Having  the  quality  program  be  in  harmony  with  the  other 
"administrative  and  technical  programs"  implies  a  breadth  of 
coverage  to  include  all  functions  within  the  contractor's 
facility,  and  an  integration  of  programs  that  fosters 
communication.  Considering  the  next  process  as  the  customer 
simplifies  the  definition  of  requirements:  anybody 
receiving  information  from  an  organization  is  defined  as 
that  organization's  customer,  and  their  requirements  must  be 
considered  when  sending  them  information.  This  requires  an 
understanding  of  the  needs  of  the  other  functional 
organizations  and  a  willingness  to  integrate  systems  and 
programs . 
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The  next  process  is  not  limited  to  organizations, 
however,  but  includes  the  floor  worker  and  the 
subcontractors  as  well.  The  quality  program  must  consider 
these  needs  as  well  as  the  needs  of  the  funtions.  The  floor 
worker  requires  unambiguous  work  instructions,  the  proper 
tools,  and  satisfactory  working  conditions  to  do  the  job 
properly.  These  requirements  make  them  the  customers  of  the 
production  engineers,  the  calibration  department,  and  the 
maintenance  department.  The  subcontractor  requires  the 
flowdown  of  quality  requirements,  the  detailed  design,  and 
the  packaging  needs.  These  requirements  make  them  the 
custome  rs  of  the  quality  department ,  the  design  engineers, 
and  the  receiving  department.  Realizing  that  the  "customer" 
can  be  anyone  encourages  the  flow  of  the  quality  program 
throughout  the  company. 

Quality  Indicators.  Formats  for  several  quality 
indicators  were  presented  in  Chapter  IV  accompanied  by  a 
limited  discussion  concerning  each  indicator.  This  section 
will  review  these  indicators  by  discussing  the  merits  and 
shortfalls  of  each  indicator. 

Defect  /  Yield.  Defect  and  yield  charts  usually 
constitute  the  largest  portion  of  the  tangible  indicators 
tracked  by  a  company  due  mostly  to  the  wide  applicability  of 
this  category.  Defects  are  usually  based  on  inspection 
results  and  can  apply  to  all  stages  of  the  production  effort 
including  receiving,  in-house  component  manufacture, 
assembly,  solder,  and  final  visual  inspection.  Yields  are 
typically  based  on  test  results  including  incoming,  printed 
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wiring  board  test  (bare  and  populated),  and  line  replaceable 
unit  (LRU)  under  a  variety  of  conditions  from  ambient  to 
freezing  to  excessive  heat. 

Defect  and  yield  charts  are  of  most  use  to  the 
first-level  quality  managers  who  can  effect  immediate 
changes  or  studies  based  on  the  results  of  data  analysis. 
Data  collection  and  analysis  must  be  timely  for  these 
indicators  to  be  useful,  supporting  the  need  for  automated 
data  collection  and  analysis.  Summary  charts  such  as  the 
Pedigree  charts  used  by  Texas  Instruments  (Figures  15  and 
16)  are  useful  as  an  informative  tool  to  keep  upper 
management  aware  of  the  current  defect  or  yield  trends, 
while  trend  charts  are  valuable  tools  for  informing  the 
quality  manager  of  the  effects  of  specific  corrective  action 
programs  (Figures  13  and  14)  and  long-term  quality 
improvement  efforts  (Figure  12). 

The  generalized  yield  chart  (Table  5),  affectionately 
known  as  a  "seeing-eye  chart"  by  the  quality  manager,  is  a 
busy  chart  potentially  containing  a  great  deal  of 
information.  A  review  of  this  chart  will  identify  those 
items  with  the  lowest  yields  and  those  processes  with  the 
lowest  yields,  allowing  the  quality  manager  to  direct  his 
attention  to  those  areas.  The  total  yield  column  is  the 
product  of  the  yields  at  each  point  through  which  an  item 
passes.  As  is  true  with  reliability  figures,  the  value  of 
the  total  yield  column  is  dependent  on  the  number  of  yield 
points  used  in  the  calculation  (e.g.,  0.9  x  0.9  »  0.81  while 
0.9  x  0.9  x  0.9  ■  0.729).  If  the  total  yield  column  is 
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given  unnecessary  emphasis,  especially  be  the  QAS ,  the 
quality  manager's  incentive  to  remove  a  point  or  two  from 
the  chart  would  be  great.  The  reader  must  keep  in  mind  that 
the  purpose  of  indicators  from  both  the  contractor's  and 
government's  point  of  view  is  the  same:  identify  those 
areas  needing  the  most  attention.  This  goal  can  be  achieved 
with  this  chart  by  emphasizing  the  proper  areas. 

Cost  of  Quality.  Cost  of  quality  information, 
including  scrap,  rework,  and  repair  (SRR),  was  indentified 
by  ASD  as  being  of  greatest  use  when  dealing  with 
contractors  who  are  reluctant  to  initiate  quality 
improvement  efforts.  Highlighting  to  a  manager  the  amount 
of  money  essentially  thrown  out  the  window  can  produce 
dramatic  results.  At  the  conscientious  quality  manager's 
level,  however,  cost  of  quality  information  is  not  as 
useful.  If  the  quality  manager  has  active  efforts  in 
reducing  and  preventing  defects,  raising  yields,  and 
narrowing  process  deviations,  the  cost  of  quality  will 
decline  accordingly.  The  charts  can,  .lowever,  identify  the 
areas  requiring  the  greatest  attention  and  the  cost  trends 
in  various  a.  'as  (Figures  2-6,  Tables  2  -  3). 

Cost  data  is  among  the  most  proprietary  data  easily 
available  to  the  QAS  and  will  be  the  first  information 
withheld  if  the  contractor  determines  that  the  information 
is  being  used  for  comparison  to  other  programs.  Due  to  the 
different  auditing  systems  and  cost  classifications,  it  is 
not  possible  to  compare  cost  data  across  programs  or 
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companies.  The  information  should  be  used  only  for  problem 
identification  and  trending  purposes  within  the  program. 

Because  cost  data  are  used  only  within  a  given  program, 
the  data  base  used  to  support  the  indicator  is  not  of  great 
importance  as  long  as  the  data  include  all  major 
contributors  to  the  costs  being  identified.  Displaying 
information  as  a  percentage  or  ratio  (Figures  2,  4,  6,  8, 

10,  and  28)  takes  into  account  the  varying  level  of  activity 
for  a  given  period,  while  displaying  absolute  values 
(Figures  3,  5,  7,  9,  and  11;  Tables  3  and  4)  identifies 
those  areas  accounting  for  the  largest  share  of  the  costs. 
Tables  and  chcts  presenting  information  based  on  both 
absolute  values  and  ratios  (Figure  27  and  Table  2)  combine 
the  benefits  of  the  previous  formats. 

Process  Control.  The  theory  supporting  use  of 
process  control  is  based  on  the  contention  that  by 
carefully  controlling  the  process  one  will  be  controlling 
the  quality  of  the  result.  Process  control  therefore  lends 
itself  to  preventative  actions  as  opposed  to  the  corrective 
reactions  inherent  in  inspection,  and  is  applicable  to 
processes  measuring  either  attributes  such  as  pass/fail  or 
discrete  values  such  as  weight  or  length.  Although  a 
description  of  process  control  is  beyond  the  scope  of  this 
study,  more  thorough  discussions  are  available  (8). 

The  nature  of  process  control  charts  dictates  that 
they  be  used  by  the  managers  and  operators  closest  to  the 
process.  An  unfavorable  trend  identified  by  the  control 
chart  needs  to  be  corrected  immediately  and  not  after  it  has 
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been  discussed  at  the  weekly  meeting.  Consequently,  the  QAS 
will  not  likely  monitor  the  control  charts  themselves  so 
much  as  he  will  monitor  the  effective  use  of  the  flow 
charts  by  the  contractor. 

Pareto  Analysis.  Quality  managers  are  very 
aware  of  the  tendancy  for  80%  of  the  problems  to  be  caused 
by  20%  of  the  parts,  giving  rise  to  the  use  of  Pareto 
analyses  (Figures  17  and  18)  as  quality  indicators. 
Especially  useful  on  defect  analysis,  Pareto  charts  can 
identify  either  the  parts  with  the  highest  defect  rates  or 
the  defects  with  the  highest  rate  of  occurance.  These 
charts  do  not  take  into  account  the  fluctuating  level  of 
activity  that  may  influence  the  number  of  defects  detected. 

Vendors .  Vendor  rating  systems  are  valuable 
tools  for  monitoring  the  quality  of  those  subcontractors 
supplying  parts  for  use  in  the  contractor's  product.  The 
vendor  rating  system  discussed  in  this  study  is  based 
primarily  or.  the  rejection  rate  of  the  supplier's  product. 
However,  automatic  rating  downgrades  are  possible  if  the 
vendor  fails  to  respond  adequately  to  corrective  action 
requests  or  fails  to  pass  a  quality  survey.  An  additional 
criteria  could  be  to  include  vendor  reliability  by  factoring 
early  or  late  deliveries  into  the  rating  formula  (early 
deliveries  may  not  be  desirable  if  the  contractor  has  no 
place  to  store  the  shipment  until  it  is  needed). 

The  indicators  presented  in  this  study  (Figure  1  and 
Table  1)  are  primarily  useful  for  informing  upper 
management  of  the  vendor  rating  status.  The  ratings  of 
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individual  vendors  are  used  by  the  quality  department  in 
determining  where  to  focus  their  attention.  Note  that  "C" 
rated  vendors  may  be  sole  suppliers  of  critical  parts  and 
termination  of  future  contracts  with  those  vendors  may  not 
be  an  option.  In  these  cases  the  quality  department  must 
work  closely  with  the  vendors  to  improve  their  reliability. 

Material  Review  Board.  Monitoring  material 
review  board  ( MRB )  actions  will  give  the  QAS  an  indication 
of  the  amount  of  nonconforming  material  being  produced  or 
received.  whether  the  company  allows  use-as-is  or 
nonstandard  repair  procedures  or  not,  any  MRB  disposition 
(standard  repair,  scrap,  rework,  or  return  to  vendor)  is  the 
result  of  nonconforming  material.  Comparison  charts  such 
as  those  used  by  ESD  (Figures  25  and  26)  are  useful  in 
determining  whether  the  program  of  interest  is  having  more 
problems  than  company  in  general. 

A  product  requiring  an  MRB  disposition  is  the  result  of 
a  process  deviation.  Perhaps  more  important  than  the 
absolute  or  relative  number  of  MRB  dispositions  is  the 
effectiveness  of  the  corrective  actions  implemented  to 
prevent  reoccurrence  of  the  problems. 

Engineering  Change  Notices .  Engineering  change 
notices  (ECNs)  are  generated  as  a  result  of  change  within 
the  program.  Considerable  change  might  be  expected  early  in 
a  program  while  design  efforts  are  being  finalized,  but 
stability  is  expected  as  the  program  matures.  The  cost  of 
ECNs  can  be  presented  as  discussed  earlier,  while  the  trend 


in  the  number  of  ECNs  generated  can  also  be  evaluated 


(Figure  19).  The  causes  of  and  functions  responsible  for 
ECNs  can  be  identified  to  indicate  areas  where  additional 
attention  may  be  required  (Table  6). 

Customer  Returns.  The  using  command  will  often 
be  interested  in  the  status  of  customer  returns,  defined 
here  to  be  the  end  user's  returns  of  product  that  has 
failed  in  the  field.  A  chart  identifying  the  number 
of  units  at  each  stage  (Figure  20)  will  identify  bottlenecks 
to  be  dealt  with  while  a  status  chart  (Figure  21)  will 
identify  the  results  of  bottleneck-clearing  efforts. 

Othe r s  .  The  remaining  indicators  identified  in 
this  study  (design,  drawing  releases,  work  instructions, 
and  purchase  order  releases)  are  additional  areas  the  QAS 
should  monitor.  These  areas  as  discussed  can  provide  signs 
of  impending  quality  trouble  but  are  likely  to  be  of 
greater  interest  to  the  manufacturing  managers.  An 
understanding  of  the  requirement  to  include  quality  in  all 
facets  of  the  company  will  encourage  the  QAS  to  stay  aware 
of  the  status  of  all  areas  of  the  company. 

Visibility  of  Indicators.  Having  visible  indicators 
does  not  necessarily  mean  posting  the  results  of  test  yields 
or  inspections  on  the  work  floor.  Although  this  is  not 
necessarily  a  bad  idea,  this  practice  might  better  be 
described  as  the  display  of  indicators  and  may  or  may  not  be 
supported  by  the  management's  quality  philosophy. 

Visibility  of  indicators  more  accurately  refers  to  the 
attention  the  indicators  receive  by  management.  The  quality 
indicators  identified  above  are  generally  designed  for  use 
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by  the  quality  department  directly  responsible  for 
monitoring  the  program.  Decisions  can  be  made  at  that  level 
concerning  the  daily  operations  based  on  the  information 
presented.  Upper  management,  both  contractor  and 
government,  must  be  kept  abreast  of  the  quality  issues, 
however.  This  attention  enforces  the  management's 
commitment  to  quality  and  may  be  giving  the  quality  manager 
the  authority  to  make  the  necessary  decisions.'  Although 
certain  formats  may  not  be  suitable  for  upper  management  due 
to  the  level  of  detail  (see  Table  5),  summary  charts 
accompanied  by  explanations  for  any  deviation  (up  o_r  down) 
are  necessary. 

Automation .  The  installation  of  computer  systems 
within  the  quality  system  can  significantly  improve  the 
effectiveness  of  the  program.  Three  areas  lend  themselves 
well  to  automation:  processes,  test  and  inspection,  and 
data  collection. 

Many  production  processes  in  the  defense  electronics 
industry  can  benefit  from  automation.  The  automated  systems 
are  inherently  more  reliable  once  the  process  has  been 
refined  to  meet  the  needs  of  the  program,  consistantly 
producing  products  within  a  narrow  tolerance  band.  Examples 
include:  processes  requiring  high  degrees  of  accuracy  such 
as  bonding  fine  wires  between  hybrid  circuits  and  the 
circuit  board,  high  volume  processes  such  as  the  soldering 
of  components  to  the  circuit  board,  and  repetitive  processes 
such  as  inserting  simple  components  (e.g.,  integrated 
circuits,  resisters,  capacitors,  and  diodes)  on  the  circuit 
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board.  Automation  of  processes  such  as  these  also  invites 
the  application  of  statistical  process  control. 

The  complexity  of  the  products  in  the  defense 
electronics  industry  in  many  cases  dictates  the  necessity  of 
automated  test  and  inspection.  Circuit  boards  with  hundreds 
of  potential  electrical  paths  are  assembled  into  line 
replaceable  units  (LRUs)  with  thousands  of  potential 
electrical  paths.  These  LRUs  must  be  evaluated  and  strpc^pH 
to  insure  survivability  in  the  final  weapon  system,  a 
formidable  task  if  required  to  be  performed  manually. 
In-circuit  testers  that  verify  the  electrical  integrity  of 
the  LRU  and  environmental  test  chambers  that  operate  LRUs  at 
simulated  environmental  extremes  are  two  examples  of 
automated  test  equipment  commonly  used. 

Just  as  automated  processes  can  reduce  the  standard 
deviation  of  precision  processes,  inspection  can  h<*lp  insure 
timely  detection  of  products  that  do  not  conform. 

Coordinate  measuring  machines  can  confirm  the  alignment  of 
critical  components,  wire-pullers  can  verify  the  bond 
strength  of  fine  wires,  and  cameras  can  detect  the  absence 
or  misalignment  of  components  on  a  circuit  board. 

Automated  data  collection  and  analysis  can  dramatically 
reduce  the  time  necessary  to  determine  the  existance  of  a 
current  or  potential  problem.  By  making  indicators  more 
"real  time,"  fewer  defective  items  will  pass  before  the 
problem  is  noted,  allowing  the  indicators  to  be  used  as 
preventative  tools  rather  than  reactive  tools.  Automation 
of  manual  data  collection  is  relatively  simple--have  the 
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deficiencies  entered  into  a  computerized  data  base  rather 
than  onto  a  piece  of  paper.  Automated  test  and  inspection 
operations  also  enter  anomolies  into  the  data  base 
automatically.  In  addition,  an  automated  data  collection 
and  analysis  system  can  enhance  the  flexibility  and 
usefulness  of  the  information  reporting  system. 

Continuous  Improvement.  The  basis  for  Total  Quality 
Management  is  continuous  improvement  in  all  aspects  of  all 
processes.  This  idea  can  easily  be  applied  to  the  aspects 
of  the  contractor's  quality  program,  but  must  be  applied 
critically.  In  areas  where  much  improvement  is  possible 
(based  on  the  experiences  cf  both  the  QAS  and  the 
contractor)  a  continuous  improvement  goal  might  not  provide 
the  incertive  necessary  to  achieve  those  large  gains.  In 
this  case,  continuous  improvement  would  imply  that  a  little 
improvement  this  year  and  a  little  improvement  next  year  is 
acceptable,  even  when  great  improvement  is  expected.  In  a 
situat.^r.  where  improvement  has  shown  to  be  slow  and 
difficult,  again  based  on  the  joint  experiences,  continuous 
improvement  might  be  an  acceptable  goal,  implying  that  any 
improvement  is  a  good  trend. 

The  QAS  should  be  aware  that  continuous  improvement  may 
be  a  utopia,  as  suggested  by  ESD .  Improvement  may  be 
limited  by  the  physical  technology  of  the  process  (e.g.,  a 
wave  solder  machine  or  a  calibration  device)  or  by  the 
current  "intellectual  technology."  The  latter  refers  to  the 
design  of  experiments  to  improve  the  process.  Taguchi's 
method  for  bringing  a  process  to  its  peak  can  reduce  the 
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number  of  experimental  runs  necessary  to  "tune"  a  process 
from  hundreds  to  a  dozen  (21:65-71).  This  advance  in 
intellectual  technology  breathed  new  life  into  the 
improvement  of  many  physical  processes. 

The  QAS  and  the  contractor  should  mutually  consider  the 
expectations  and  technical  capabilities  of  a  process  in 
light  of  the  current  intellectual  technologies  and  set 
realistic  goals  for  that  process. 

Chanaina  Phi  lncnr>^ioSi  Traditional  views  ul  cue 
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customer  are  as  the  user  of  the  final  product.  In  the 
defense  electronics  industry,  this  is  defined  as  the  airman 
or  soldier  using  the  end  item.  This  definition  proved 
difficult  to  work  with  due  to  the  customer  often  being 
thousands  of  miles  from  the  contractor.  Feedback  about  the 
usefulness  of  the  product  was  filtered  through  several 
layers  of  command  before  reaching  the  manufacturer,  forcing 
at  least  a  delay  in  production  modifications  and  at  worst  a 
requirement  for  field  retrofit.  Consequently,  the 
traditional  contractor  has  developed  his  quality  programs 
around  the  idea  of  conformance  to  requirements.  Since  the 
customer  was  1  t  easily  accessible,  the  specifications  were 
used  as  a  subset  ate. 

This  study  has  discussed  the  shift  in  philosophy  away 
from  conformance  to  requirements  and  towards  fitness  for 
use.  This  shift  has  been  made  possible  and  beneficial  by 
the  redefinition  of  customer  from  "end  user  only"  to  "next 
process,"  keeping  in  mind  the  possibility  that  a  product 
satisfying  the  next  process  may  not  satisfy  the  ultimate 
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"end  user."  While  a  fitness  for  use  philosophy  does  not 
disreyard  the  contract  '■oquirements,  it  does  require  the 
additional  consideration  of  requirements  not  found  in  the 
speci f ication  or  company  policy.  The  result  of  these 
additional  considerations  may  be  the  revision  of  policy,  but 
more  importantly,  is  the  completion  of  a  job  the  right  way 
the  first  time.  The  end  user  is  not  forgotten  in  the 
revised  philosophy,  however,  supporting  the  need  for 
monitoring  customer  returns. 

Barriers 

The  discussions  in  the  previous  section  centered  on 
what  an  "ideal"  quality  pruyram  might  include.  There  are, 
however,  Lwo  major  barriers  to  developing  an  ideal  quality 
program:  financial  and  technological.  The  discussion  of 

financial  barriers  will  focus  on  the  profit  structure  of  the 
typical  production  contract,  while  the  review  of 
technological  barriers  will  center  on  short  production  runs, 
complex  products,  and  che  gray  areas  of  inspection. 

Financial  Barriers.  Proponents  of  preventative  quality 
assurance  insist  that  quality  is  free,  meaning  that  any 
money  spent  on  quality  assurance  can  repay  the  contractor 
fully  in  the  improved  quality  of  the  product  and  reduced 
production  costs.  This  idea  does  not  consider  the 
contractual  implications  of  cost  reduction.  The  profit  on 
firm,  fixed-price  contracts  typically  used  for  production 
efforts  has  been  traditionally  computed  as  a  percentage  of 
the  contract  cost.  Any  cost  reductions  during  the 
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performance  of  that  contract  benefits  the  contractor 
directly  in  the  form  of  higher  profits.  Contracts  for 
additional  buys  of  the  same  product  will  benefit  the 
government  in  the  form  of  reduced  cost  (due  to  the  process 
cost  reductions)  but  lower  the  profit  available  to  the 
contractor.  As  more  quality  improvements  are  made,  the 
contractor's  profit  declines,  a  system  that  does  not  provide 
the  proper  incentive  for  the  contractor  to  investigate  cost 
reduction  techniques. 

This  barrier  has  been  recognized  and  experimental 
contracting  efforts  are  being  implemented  to  study  different 
applications  of  profit. 

Technological  Barriers.  Technological  barriers  to 
implementing  an  ideal  quality  program  also  confront  the 
contractor.  Among  these  are  short  production  runs,  product 
complexity,  and  the  gray  areas  inherent  in  inspection. 

Fiscal  constraints  limit  the  number  of  systems 
procured.  Total  production  runs  can  be  in  the  single  digits 
for  satelite  systems  or  as  many  as  several  hundred  for 
aircraft  "black  boxes."  Large  buys  are  often  broken  into 
smaller  lots  due  to  the  uncertainty  of  government  funding. 
These  short  production  runs  and  uncertainty  of  future  lots 
limit  the  ability  of  the  contractor  to  develop  a  truly 
mature  program.  Breaks  in  the  production  line  impact  the 
learning  curve,  while  periods  of  low  production  inhibit  the 
collection  of  data  for  trend  analysis. 

Compounding  the  problem  caused  by  short  production  runs 
is  the  complexity  of  the  hardware  being  produced.  Production 
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difficulties  on  simple  products  can  be  detected  and  resolved 
after  the  production  of  only  a  few  items;  detecting, 
resolving,  and  preventing  inconsistent  problems  on  complex 
LRUs  is  a  much  more  involved  process  requiring  extensive 
data  analysis.  Production  processes  involving  complex 
hardware  are  slow  to  mature,  resulting  in  a  slowly  maturing 
quality  program  (when  viewing  a  mature  quality  program  as 
being  more  preventative  than  r ea c t i ona ry ) . 

Gray  areas  of  inspection,  discussed  in  the  section 
covering  ITT's  observations,  are  the  third  area  hindering 
development  of  an  ideal  quality  program.  Two  inspectors 
having  different  views  on  the  acceptability  of  a  "defect" 
may  categorize  that  defect  accordingly,  causing  the 
occurance  of  that  defect  to  appear  intermitant.  Intermitant 
problems  are  difficult  to  analyze,  resolve,  and  prevent, 
again  resulting  in  a  slowly  maturing  quality  program. 

Summary 

This  chapter  has  discussed  the  findings  and 
observations  obtained  during  the  study.  The  issues  raised 
in  the  early  chapters  have  been  reviewed,  and  some  of  the 
problems  facing  the  new  Quality  Assurance  Specialist  have 
been  addressed.  Elements  that  an  "ideal"  quality  program 
might  contain  have  been  identified,  including  the  necessity 
of  proper  quality  management,  examples  of  quality  indicators 
and  need  for  visibility,  the  applications  of  automation,  the 
issues  involved  with  continuous  improvement,  the  shift  in 
quality  philosophy,  and  the  barriers  to  the  development  of 
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an  ideal  quality  program.  Solutions  to  some  of  the  issues 
raised  are  beyond  the  scope  of  this  study,  but  they  are 
mentioned  as  information  for  the  QAS . 

The  QAS  is  should  to  review  the  information  contained 
in  this  study  and  apply  the  indicators  and  ideas  judiciously 
to  his  program,  remembering  that  quality  innovation  has 
resulted  in  many  of  the  indicators  presented. 
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VI .  Recommendations  for  Further  Study 


This  thesis  reviewed  quality  indicators  and 
observations  from  the  defense  electronics  industry  and  the 
government  agencies  working  with  that  industry.  The 
results  were  presented  in  the  form  of  discussions 
focusing  on  indicators  to  be  used  by  the  new  Quality 
Assurance  Specialist  in  developing  a  set  of  indicators  for 
monitoring  a  contractor's  quality  program.  Several 
recommendations  are  offered  for  continuing  this  study. 

A  limited  population  was  chosen  for  this  study  due  to 
time  and  fiscal  constraints.  Although  the  two  contractors 
studied  have  excellent  quality  programs,  they  still  result 
in  only  two  data  points.  On  the  government  side,  quality 
managers  in  staff  positions  were  interviewed  resulting  in 
more  philosophical  views  of  quality  than  might  be  presented 
by  an  experienced  QAS  still  interfacing  with  contractors  and 
monitoring  quality  indicators.  Further  study  should  include 
r , uuu a  v^vjiiL*.ouv.oro  ci i i government  quality  assurance 
specialists  still  "in  the  trenches." 

This  study  focused  on  the  quality  indicators  used 
primarily  by  the  managers  within  th®  qualify  den^rtment 
itself.  Considerable  emphasis,  however,  was  placed  on  the 
necessity  of  a  comprehensive,  integrated  quality  effort 
throughout  the  company.  Future  studies  could  evaluate  the 
quality  indicators  used  within  other  functional  offices. 

The  quality  indicators  identified  in  this  study  were 
tangible  indicators.  Tangible  indicators  present  the 
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advantage  of  being  relatively  easy  tc  define,  measure,  and 
report.  They  do  not,  however,  encompass  the  entire  quality 
program,  excluding  such  elements  as  quality  improvement 
programs  and  management  commitment.  An  additional  study 
could  be  accomplished  focusing  on  ways  to  measure  and  report 
these  intangible  elements. 

Although  quality  assurance  indicators  used  in  the 
defense  electronics  industry  were  identified  in  this  study, 
giving  the  QAS  some  tools  for  use  in  his  "toolbox”  of 
quality  indicators,  no  direct  attempt  was  made  to  suggest 
methods  of  improving  the  current  situation  faced  by  the  new 
QAS.  Further  study  into  this  area  could  include  the 
development  of  a  QAS  education  and  training  model  that  would 
identify  the  required  courses,  training,  and  professional 
reading  lists  beneficial  to  the  development  of  a  well- 
rounded  QAS  . 

Finally,  future  studies  could  focus  on  the  technical 
development  of  the  QAS.  The  QAS  is  not  a  technical  expert 
when  first  entering  this  field  and  needs  to  know  not  only 
what  to  look  for  (such  as  the  indicators  identified  in  this 
study),  but  also  how  to  use  that  information.  Examples  of 
specific  indicators  could  be  presented  showing  how  to 
identify  good  and  bad  situations  or  trends. 
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